General Disclaimer 


One or more of the Following Statements may affect this Document 


• This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 


• This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 


• This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 


• This document is paginated as submitted by the original source. 


• Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 


Produced by the NASA Center for Aerospace Information (CASI) 



* 

< 




-> •• • 


1 1 


» * 
* ^ 

r (NASA-TH-85146) EVALUATION 07 

MANAGEMENT 


N83-1 3036 ^ 

MEASURES OP SOFTMABE DEVELOP BE NT. 

VOLUME 1: 



ANALYSIS SUh2»BY (NASA) 142 p 

HC 

A07/MF AO 1 





CSC l 09B 


U acids 

L. 


G3/6 1 

02142 j 

/x • : 

► 

» 

• 

4 

<■ 

■ \ . > 

. . i • . 

V • 

♦ 

0 

* 

• 

' % * • 

.. V 1 : •' 

• »/ 


•> 

. ' 4 

* m 

0 « 

• 

i 

« 


« 

« 

» ' *v. 


• 


4 

\ 


hr 


.v '• 



I ' . 






# 


\ 


& . 







SOFTWARE ENGINEERING LABORATORY SERIES 


SEL-82-001 


8 


EVALUATION OF MANAGEMENT 
MEASURES OF SOFTWARE 
DEVELOPMENT 

VOLUME 1: ANALYSIS SUMMARY 


SEPTEMBER 1982 I 


NASA— 

National Aeronautics and 
Space Administration 

Goddard Space Flight Center 

Greenbe't Maryland 20771 


FOREWORD 


The Software Engineering Laboratory (SEL) is an organization 
sponsored by the National Aeronautics and Space Administra- 
tion, Goddard Space Flight Center (NASA/GSFC) and created 
for the purpose of investigating the effectiveness of soft- 
ware engineering technologies when applied to the develop- 
ment of applications software. The SEL was created in 1977 
and has three primary organizational members: 


NASA/GSFC (Systems Development and Analysis Branch) 

The University of Maryland (Computer Sciences Department) 
Computer Sciences Corporation (Flight Systems Operation) 


The goals of the SEL are (1) to understand the software de- 
velopment process in the GSFC environment; (2) to measure 
the effect of various methodologies, tools, and models on 
this process; and (3) to identify and then to apply success- 
ful development practices. The activities, findings, and 
recommendations of the SEL are recorded in the Software En- 
gineering Laboratory Series, a continuing series of reports 
that includes this document* A version of this document was 
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also issued as Computer Sciences Corporation document 
CSC/TM- 8 2/606 3 . 


The contributors to this document include 
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ABSTRACT 


v) 


This document reports the results of an evaluation of a 
large set of software development measures relevant to the 
Goddard Space Plight Center (GSFC) environment. Volume 1 
explains the conceptual model, the data classification 
scheme, and the analytic procedures. This volume summarizes 
the analytic results and recommends specific software meas- 
ures for collection and monitoring. Volume 1 also repro- 
duces in full the results of the computer analyses. 

Volume 2 presents a detailed description of the data ana- 
lyzed including definitions of measures, lists of values, 
and summary statistics. 
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SECTION 1 - INTRODUCTION % 

This two-volume report presents the results of an analysis 
of a large set of management measures of the software devel- 
opment process. The purposes of the analysis are to charac- 
terize the current software development process in one 
environment by identifying important qualities and corre- 
sponding measures and to evaluate the effectiveness of spe- 
cific tools and techniques in this environment. The 
measures studied are counts/ ratios, and management-supplied 
ratings of various elements of the software development 
process. The measures are high level in that each describes 
some aspect of an entire software project (or a large part 
of it) rather than individual components of the project. 

The data analyzed have been collected by the Goddard Space 
Plight Center (GSFC) Software Engineering Laboratory (SEL) 
from 25 actual medium-scale, scientific software projects 
developed for flight dynamics applications. .Values have 
been determined for over 600 measures for each of the proj- 
ects studied. Several different statistical procedures have 
been employed to investigate these measures. 

This document describes the following aspects of this re- 
search effort: 

• Motivation, rationale# and objectives 

• Source, nature, and derivation of the measures 
Analytic procedures employed 

• Results of the analysis 

• Identification of specific measures useful to the 
management of software development activities 

The data, procedures/ and results are summarized in the 
text. Appendix A of this volume reproduces the results of 
computer -generated factor analyses of the data. Appendix A 
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of Volume 2 contains a detailed description of the data in- 
cluding identification of measures# lists of values# and 
summary statistics. Sources cited in the references provide 
additional useful material. 

1.1 CONCEPT OF MEASURES 

The need to measure the quantity and quality of developed 
software is self-evident. Measures of productivity, relia- 
bility# size# and complexity# for example# are vital to 
software planning and management. What is not so evident# 
however# is which are the most important quantities and 
qualities and what are the best measures of them. One ap- 
proach that the SEL has taken to resolve these questions has 
been to gather as many measures as possible and to systemat- 
ically evaluate their utility. This report documents that 
approach. 

Measures should be distinguished from qualities. As the 
term is used here# a measure is a count or a numerical rat- 
ing of the occurrence of some property. Examples of meas- 
ures include lines of code# number of computer runs# 
person-hours expended# and degree of use of top-down design 
methodology. A quality is a high-level characteristic to 
which one or more measures may be related. For example, the 
measures of errors per line of code and mean time to failure 
are related to the quality of reliability. However# neither 
measure alone adequately quantifies reliability. 

Measures appeal both to the researcher and the manager as 
potential means of defining# explaining, and predicting 
software development qualities# especially productivity and 
reliability. These goals can be realized most efficiently 
by developing a single effective measure for each quality of 
interest. That is one of the purposes of this analysis. 

Measures may be classified into four groups as illustrated 
by the software development model presented in Figure 1-1. 
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It shows these components: a problem# a solution-generating 

process, the environment in which that process takes place# 
and the solution (or software product). Measures can be 
used to characterize the components of this model and to 
show their interrelationships. For example, resource utili- 
zation measures record the rate at which resources in the 
environment (especially personnel and computer resources) 
are used by the software development process. Figure 1-1 
also shews other examples of appropriate measures for each 
component. 

Measures may be further characterized as subjective or ob- 
jective . The exercise of human judgment in assigning a 
value for a measure trikes that measure subjective* The most 
widely accepted and widely used measures of software devel- 
opment are objective measures of the software product. How- 
ever, currently available objective measures do not take 
into account the effects of development constraints and 
practices on the quality of the software product. 

Evaluation of software quality is, at present, a matter of 
the subjective interpretation of results relative to re- 
quirements. Few objective measures of software quality are 
available. Thus, the SEL has developed a set of subjective 
(or interpreted) measures that complement the more commonly 
employed objective measures of software development. The 
analysis described in this report attempts to validate those 
measures. Section 2 discusses the specific measures inves- 
tigated by the SEL; these measures describe all facets of 
, ■■■/): 
flight dynamics software development. ' 

1. 2 OBJECTIVES OF THE ANALYS I S 

One objective, then, of this analysis is to identify the 
significant software development attributes (qualities) and 
their corresponding measures from amon« those measures col- 
lected by the SEL. The other objective is to define their 


1 J 

applicability to software development management* The defi- 
nition of effective software development measures enables 
planners and managers to 

• Recognize the characteristics of problem software 
early in development 

• Identify the most effective development and manage- 
ment tools and techniques 

• Estimate the costs and output of software develop- 
ment efforts 

Section 5 recommends some specific measures to be used in 
these applications. 

1.3 RELATED RESEARCH 

A number of objective measures of software development at- 
tributes are widely accepted in the software engineering 
community. Lines of code and staff-hours are examples. 
However , measures have been refined to the point of be-» 
coming useful management tools. Basili presents a survey of 
such efforts in Reference 1. 

Subjective measures of software quality were among the first 
to be developed and actually applied to software manage- 
ment. The concepts of module strength and coupling (Refer- 
ence 2) are examples of early work in this area. 
Unfortunately, these qualities have proved difficult to 
quantify, although a rating scheme has been developed based 
on the types of cohesion and dependencies; that program mod- 
ules may exhibit. Taking another approach, McCall (Refer- 
ence 3) has developed a comprehensive system of software 
qualities and appropriate measures that others are still * 
refining and extending (Reference 4), These investigations 
include both subjective and objective measures. 

The approach to software measurement adopted for the analysis 
here is different from that generally followed. The usual 
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procedure is to specify high-level "qualities" and then to 
seek numerical criteria or measures of those qualities. How- 
ever , the approach followed here is to identify the qualities 
being measured by the data collected rather than to attempt 
to associate measures with previously specified qualities. 

A Subset of this data has been analyzed previously using 
this approach. The results of that analysis were reported 
at the Sixth Annual Software Engineering Workshop in 1981 
(Reference 5). This analysis extends the scope of that work. 


SECTION 2 - DATA DESCRIPTION 

This section describes the sources , nature , and derivation 
of measures used in the analysis. Specifically, the topics 
discussed are 

• Source of the data studied 

• Classes of measures included 

e Methods of measurement used 

A brief description of each individual measure, lists of the 
actual data, and summary statistics e^e included in Appen- 
dix A of Volume 2. 

2.1 SOURCE OF DATA 

The SEL, a cooperative effort of the Goddard Space Flight 
Center, Computet Sciences Corporation (CSC), and the Univer- 
sity of Maryland, assembled the data set used in this analy- 
sis. The SEL collects and analyzes data from software 
development projects that support flight dynamics activi- 
ties. The principal objective of the SEL is to identify and 
apply software development tools and techniques that improve 
the quality of the software development process. Refer- 
ence 6 describes the organization, operation, and accom- 
plishments of the SEL in more detail. 

The SEL has monitored the development of 43 software proj- 
ects during the past 5 years. This document analyzes a se- 
lected subset of 25 projects. The selection criteria were 
intended to strengthen the rigor of the analysis. The proj- 
ects selected were developed in the same programming lan- 
guage (FORTRAN) and used the same set of computers for 
similar and/or related applications. 

The specific type of application software studied supports 
ground-based spacecraft attitude determination and control. 
The subsystems included in a typical attitude system are 


telemetry processing# sensor calibration, attitude computa- 
tion, and maneuver planning. Table 2-1 shows some of the 
characteristics of the software studied. 


Table 2-1. Summary of 11 Mission Projects Studied 


Measure 


Developed Lines of Code (Thousands) 
Lines Developed Per Staff-Month 
Total Effort (Staff-Months) 

Average Staffing Level 
Duration (Months) 

Percentage of Effort 
Programmer 
Manager 
Other 


Median 


Interquartile 

Range 1 


Value is one-half of the difference between the third 
quartile and the first quartile. 


The 25 software development efforts selected for study have 
been combined in two different ways for purposes of compari- 
son. They are grouped into 11 mission projects composed of 
related efforts. The projects combined in this manner were 
undertaken in support of the same mission. Separately, 

20 independent software systems are identified among the 
group of 25. These are subdivided into a class of 11 large 
systems (more than 30,000 lines of code) and a class of 
9 small systems (fewer than 30,000 lines of code). Thus, 
four data groups are defined for the analysis. Appendix A 
of Volume 2 includes summaries of these groups. 




2.2 CLASSES OF MEASURES 

More than 600 measures have been assembled for this analy- 
sis. Some were suggested by other researchers. The meas- 
ures are organized into seven topical classes: 

1. Software engineering practices-- tools* techniques* 
and practices employed by the software development 
team 

2. Development team ability--quality and performance 
of tihe development team 

3. Difficulty of project--problems encountered with 
complexity* staffing* and support 

4. Process and product characteristics--evaluations of 
process performance and product quality 

5. Development team background--previous experience of 
the development team 

6. Models — measures used in some popular resource/size 
estimation models 

7. Additional detail — other objective measures of the 
software product and resources 

Table 2-2 shows the further division of the classes into 
subclasses (categories) . Some additional measures have been 
constructed by forming weighted sums of other measures. 

The measures included in these classes fully describe the 
process and product components in the software development 
model as experienced by the SEL (see Figure 1-1) . Sec- 
tion 2.1 points out that the projects studied were chosen so 
that the other components of the model* the software problem 
(application) and the development environment (computer) * 
would be as similar as possible. Thus* consideration of 
these components (problem and environment) can be minimized 
in the analysis. . -- :• 



Table 2-2. Classes of Measures 


No. of 


Sums of 


Name of Class 


Software Engineering Practices 

Practices and Techniques 
Tools 

Documentation 

Development Team Ability 

Experience With Application 
Effectiveness of Management 
Performance of Team 

Difficulty of Project 

Complexity of Problem 
Internal Influences on 
Project 

External Influences on 
Project 

Process and Product Character* 
istics 

Resources Available 
Software Product 
Product/Process Performance 

Development Team Background 

Team Rank 

Years of Professional 
Experience 
Years of Applicable 
Experience 
Years of Environment 
Experience 

Resource Model Parameters 

Walston-Felix 
PRICE S3 
COCOMO 

Additional Detail 

Miscellaneous 
Code Breakdown 
Estimated Statistics 


Symbol 

Measures 

Measui 

■ v : ' ! ' ' 

SE 

0 

1 

MT 

30 

4 

TS 

\ 15 

1 

DC 

15 

1 

AB 

0 

12 

AP 

15 

4 

MG 

35 

13 

PF 

40 

0 

DF 

0 

1 

CP 

15 

5 

IN 

15 

v 4 

EX 

20 

7 

RA 

20 

5 



Weighted sums of other measures 
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2.3 METHODS OF MEASUREMENT j 

Measurement is the process o£ assigning a number or a state 
to represent a quantity or quality. Two general methods of 
measurement exist: objective and subjective. This analysis 

includes both types of measures. 

Objective measures are often the result of counting proc- 
esses; these are measures of tangible physical quantities 
and qualities. Examples include lines of coder staff -hours, 
and computer language used. Some of the objective measures 
collected by the SEL have been restated as relative scores 
determined from defined ranges of values. 

Subjective measures result from classification or rating 
processes involving the exercise of human judgment. How- 
ever, the evaluator also uses objective measures as guide- 
lines in assigning values for some of the subjective quality 
measures developed by the SEL. The values of the subjective 
measures employed in this analysis are expressed as relative 
scores on a scale from 0 to 50. Table 2-3 shows the inter- 
pretation of these scores. 

Table 2-3. Interpretation of Subjective Measures 


Score 

Approximate 

Percentage 

Interpretation 

0 

0 

Never , none 

10 

20 

Rarely, very poor 

20 

40 

Occasionally, poor 

30 

60 

Frequently, good 

40 

80 

Usually, very good 

50 

100 

Always, excellent 


Table 2-4 shows an example of how data composed of this type 
of measure might look. This technique is used extensively 
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to rafce the degree of use of various methodologies , tools, 
and management practices* The values for these measures are 
determined for each project after its completion or after 
completion of a major phase of development. This determina- 
tion occurs during a detailed review by management personnel 

involved in /'the development effort. 

. ■■ | ■ . . ; ; " 

Table 2-4. Example?’ of Subjective Measures 


Project 

MEASURE1 

MEASURE2 

MEASURES 

MEASURE4 

1 

0 

30 

20 

50 

2 

10 

10 

40 

20 

3 

•. y/ ■ 

50 

20 

30 

0 

4 ■ f 

20 

40 

10 

30 

5 f: ; 

10 

50 

50 

20 

6 

30 

20 

10 r 

40 

7 i 

30 

0 

20 

10 

8 

50 

40 

20 

0 

9 

10 

50 

0 

30 

10 

20 

0 

30 

40 


0 

B 

Q 

11 

0 

0 

0 

r\ 

o 

0 


n 


1 


Illustrative values for hypothetical measures. 
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SECTION 3 - ANALYTIC PROCEDURE 


The measures described in Section 2 are not necessarily 
unique or independent. Some may, in fact, measure the same 
or related qualities. This analytic procedure attempts to 
identify the most basic set of qualities (or properties) 
being measured by the entire set. A basic quality is one 
that is independent of all other such qualities. This sub- 
set, then, identifies the basic quality characteristics de- 
scribing the projects from which the data have been 
obtained. Studying the relationships among these basic 
qualities provides useful insight into how the software de- 
velopment process can be improved. 


The procedure to be proposed is large scale. That is, the 
procedure is appropriate when large numbers of measures (or 
variables) are to be evaluated. The researcher interested 
in studying the relationships of only a few specific meas- 
ures can probably get better results from regression- and 
hypothesis- testing techniques. Nevertheless, this procedure 
can also be useful as a screening tool for detecting con- 
founding effects in the data before selecting other statis- 
tical techniques. 

The analytic procedure followed in this experiment has sev- 
eral steps, as illustrated in Figure 3-1. These steps are 
the application of a test of normality to screen the candi- 
date measures (data) , followed by factor analyses of those 
not rejected by the test. Analyses are performed independ- 
ently for each class of measures defined in Section 2.2; 
then, the results are combined. Graphic illustrations of 
the similarities established among the projects studied are 
produced by cluster analysis * Comparing the pattern of sim- 
ilarity based on the original set of measures with the pat- 
tern of similarity based on the basic qualities identified 
by the factor analysis steps confirms the interpretation and 
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application of these quality measures in real*world situa- 
tions. Figure 3-1 also identifies ^tables and figures that 
illustrate the data at various positions in the analytic 

" ' '%V. - - 

procedure. 

The final result of this procedure is a descriptive, rather 
than a predictive, model of the data. The procedure identi- 
fies the descriptive factors common to the set of measures. 
Thus, the original measures are organized into a number of 
groups (each corresponding to a factor) smaller than the 
number of measures input to the procedure. These factors 
correspond to the basic qualities sought for in the data* 

The statistical bases of the steps in this procedure are 
discussed in more detail in the following sections. The 
statistical software used throughout the analysis is the 
Statistical Analysis System (Reference 7). 

3.1 TEST OF NORMALITY j 

The test of normality analyzes the probability distribution 
of the values of a measure. The factor analysis procedure 
is based on the assumption that the values of the measures 
input to it are normally distributed. In practice, any ap- 
proximately symmetrical distribution may be processed with- 
out seriously perturbing the results (Reference 8) . 

However, asymmetric (or skewed) data distributions can pro- 
duce misleading results* They are detected by the test used 
in this analysis. 

Figure 3-2 shows both the normal distribution and a skewed 
distribution. Because the values of the subjective measures 
are relative scores, skewed distributions result for degree- 
of-use measures when there are few examples of use in the 
data. Consequently, most projects have scores of zero for 
these measures, producing dramatically skewed distribu- 
tions. A M t M statistic and the 0.01 significance level are 




used to test the hypothesis (foe each measure) that the mean 
of the distribution is zero. 

Accepting this hypothesis is equivalent to concluding that 
the distribution is skewed because zero is a limit of the 
range of values. That is, no values less than zero are pos- 
sible for these measures. Section 4.1 reports the results 
of the test of normality. 

3.2 FACTOR ANALYSIS 

The goal of factor analysis is to discover the underlying 
structure of the data. Factor analysis hypothesizes the 
existence of a set of statistically independent "factors" 
that cannot be directly measured by the experimenter. Meas- 
ures (or variables) are the quantities that are observed in 
practice. However, the apparent correlations among measures 
can be interpreted to be caused by their joint correlation 
with common factors (see Figure 3-3). That is, two or more 
measures correlated with the same factor are correlated with 
each other. The desirable result of a factor analysis is 
the extraction of a smaller set of factors whose relation- .« 
ships are known (they are independent) from the larger set 
of measures whose relationships are more complex. 

Consider this example of the factor concept. The number of 
errors in a piece of software and its mean time to failure 
are measures related to reliability and are correlated with 
each other. However, neither measure by itself is a full 
description of reliability. Such things as the location of 
the error and the severity of the failure must also be con- 
sidered* Therefore, the reliability quality factor is not 

■ o . . 

directly measurable, although a number of measurable vari- 
ables are correlated with it. 

A successful factor analysis explains such groups of related 
measures. Thus, each factor defined corresponds to a dis- 
tinct basic quality being measured by the original set of 
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measures* These qualities are the sources of variation (or 
differentiation) in the measures among the projects studied; 
therefore# these qualities form the basis for comparisons 
among the projects. 

The output of the factor analysis procedure includes three 
types of information that are essential to interpreting its 
results. These information types are 

• Factor loading-percentage of the variance in the 
data accounted for by each factor: shows the rela- 
tive importance of factors 

e Factor pattern — correlations of all measures with 
all factors; shows the underlying structure of the 
data 

e Communality— percentage of each measure's variance 

accounted for by all factors; shows how well each 
measure is explained by the factor n'odel 

Sections 4.2 and 4.3 report the results of the factor analy- 
ses of the SEL measures. The principles of factor analysis 
are explained in more detail by Harman (Reference 9) .... 

3.3 CLUSTER ANALYSIS 

The technique of cluster analysis (or numerical taxonomy) is 
an objective method of defining and displaying groups, or 
clusters, of objects (projects, in this case) that are simi- 
lar with respect to a specified set of measures. The degree 
of similarity is determined by calculating a Euclidean dis- 
tance measure from the measures input to the procedure. 

A simple example clarifies this concept. Consider a set of 
three measures applied to each of several projects. Each 
project can be represented as a point in three-dimensional 
space where each of the three measures forms an axis (or 
dimension) as illustrated in Figure 3-4. 
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The Euclidean formula for the distanca between two points in 
a three-dimensional apace ia aa follows* 

p , ■ . ; ~ , . / x 

Do- /(X* - x M )* + (y' - y") 2 + <** - * M ) 2 

' •; - ' Ct . 

whara D - distance 

x' # y'# z' - coordinates of point 1 r 

x" r y H r z M » coordinates of point 2 | 

’ ■“ Y\ 

This formula can be easily extended to the general case of n 
measures or dimensions. Thus, the distances between all 
pairs of points can be determined and any clusters can then 
be identified. The results are presented in a display# or 
map# showing the similarities of the projects. Section 4.3 
summarizes the results of the cluster analyses of sel data. 
Reference 10 explains the principles of cluster analysis in 
more detail. 
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SECTION 4 - EVALUATION OF MEASURES 

This section presents the results of applying the analytic 
procedure (described in Section 3) to the SEL data (de- 
scribed in Section 2). Data screening, the factor analyses 
of classes of measures, and the summary analysis of factors 
are discussed. Figure 3-1 shows the sequence of these steps 
and identifies the data sets used by each. The four major 
data sets are the initial measures, screened measures, com- 
bined (class) factors; and high-level factors. 

4.1 DATA SCREENING 

A test of normality (see Section 3.1) was employed to screen 
the data (initial measures) before other analyses. The test 
rejected 32 measures. These are measures for which all 
projects were assigned the same value or measures for which 
there were too few different values. Table 4-1 lists the 
rejected measures. As indicated in the table, several of 
the measures proposed by Walston and Felix (Reference 11) or 
used in the PRICE S3 model (Reference 12} are not useful for 
characterizing the software studied by the SEL because they 
are constant from project to project. Of the 640 measures, 
608 were retained; these comprise the screened measures data 
set. 

Table 4-1. Measures Excluded From Analysis (1 of 3) 

Measure Rejection 

Code Criterion 3 

MT08 0.327 

MT27 0.046 

a Any value greater than 0.01 results in rejection. 



Use of Hierarchical Input Process- 
ing Output (HIPO) charts for design 


Presence of an independent verifi- 
cation and integration team 



0.079 


MT28 


Degree to which development team 
cooperated with and responded to 
findings of the independent verifi- 
cation and integration team 


TS04 

0.047 

Use of an online requirements* lan- 
guage tool for analysis and assess- 
ment of changes 

TS12 

0.088 

Use of an online configuration 
analysis tool for tracking) develop- 
ment activity 

DC06 

0.083 

Use of unit development folders for 
recording, in a central repository, 
development plans, status, and 
products 

AP01 

0.185 

Contribution from expert 1 

AP02 

b 

Contribution from expert 2 

IN03 

0.161 

Need for weekend overtime conditions 

IN04 

0.013 

Staffing problems during design 

INI 2 

0.073 

Project leader turnover, i.e., re- 
placement 

EX08 

0.062 

Development of front-end processors 
by an external group 

EX09 

0.016 1 

Ontime delivery of software by an 
external group 

RAO 8 

0.020 

Availability of the primary devel- 
opment computer 

RA09 

0.187 

Availability of a tertiary develop- 
ment computer 

RA12 

0.Q22 

Availability of an online process- 
ing system 

RA1 4 

0.035 

Availability of a convenient, un- 
scheduled graphic device 

‘Any value 

greater than 

0.01 results in rejection. 


All projects show the same value for the measure. 




Table 4-1. Measures Excluded From Analysis (3 of 3) 


Measure 

Code 

Rejection 

Criterion 


RA18 

0.034 

WF01 

b 

WF17 

b 

WF24 

b 

WF25 

b 

WF33 

■ I)'- 

0.171 

I 

WF38 / 

0.172 

WF48 

b 

WF49 

b 

WF64 

b 

WF65 

b 

PS 17 

b 

PS 18 

b 

PS 19 

b 

PS 20 

b 


Measure Description 


Availability of an independent test- 
ing group 

Programmer experience with the ap- 
plication 

Complexity of product's external 
communication 

Complexity resulting from hardware 
development 

Complexity resulting from security 
environment 

.1 ■ ----- 

Percentage of development in terti- 
ary compute r envi ronment 

Percentage of development in open 
computing environment, i.e., hands- 
on 1 development 

Percentage of effort for analysts 

Percentage of effort for operators 

Percentage of code for fallback and 
recovery 

Percentage of code in "other" cate- 
gory 

Code instruction mix rating 

Personnel skill mix rating 

Fraction of storage and timing ca- 
pacity 

Strictness of development standards 


l Any, value greater than 0.01 results in rejection. 


^Vll projects show the same value for the measure, 
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4.2 FACTORS WITHIN CLASSES OF MEASURES 

After screening the initial set of measures , researchers 
performed a factor analysis for each of the seven classes of 
measures defined in Section 2.2. Appendix A of Volume 1 
reproduces the results of those analyses. Table 4*2 summa- 
rizes some important features of these analyses. 

The table shows that in all cases five or six factors were 
able to explain at least 73 percent of the variance of the 
class of measures under consideration. The best case is 
that of the Development Team Background class of measures in 
which 5 factors accounted for 86 percent of the variance (or 
information content) of the 144 measures in that class. The 
38 factors produced by the 7 factor analyses form the com- 
bined factors data set. 

These results demonstrate that by focusing on factors the 
number of measurements collected during software development 
can be significantly reduced without much loss of informa- 
tion. Section 5 suggests the measures corresponding to 
these factors that can be most easily and advantageously 
collected. 

The following subsections describe the results of the factor 
analyses of classes of measures in more detail. Each factor 
is named , and correlated measures are described. Correla- 
tions between factors and measures of less than 0.56 are not 
considered in the discussion except when they form part of a 
pattern. This value is the boundary of the 0.01 signifi- 
cance region for a correlation coefficient from a sample of 
size 20. A measure having a correlation with a factor of 
0.561 or greater is termed "strongly" correlated with that 
factor. "Moderately" correlated refers to correlations 
greater than or equal to 0.444 (0.05 significance region) 
but less than 0.561. 
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Table 4-2. Summary of Factor Analyses of Classes 
of Measures 


Class of Measures 

No. of 
Measures 

No. of 
Factors 

Percentage 

of 

Variance 

Detail 

Table 

ed 

1:-: 

Software Engineering 

43 

5 

80 

A"1 


Practices V , ' 
Development Team 

110 

6 

82; ;• 

A- 2 

u ; 

Ability 

Difficulty of Project 

54 

5 

74 

A- 3 


Process and Product 

47 

5 

85 

A- 4 


Characteristics 
Development Team 

144 

5 

86 

A- 5 


Background 
Resource Mode yl l 

73 

6 

73 

A- 6 

1 

Parameters 
Additional Detail 

137 

■ 

: 6 

83 

A-7 
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4.2.1 SOFTWARE ENGINEERING CLASS 


The factor analysis of the Software Engineering class of 
measures produced a model containing five factors, which 
explain 80 percent of the variance of the class. Together 
they describe the general level of software engineering 
technology used during development and four other independ- 
ent subgroups of technologies. Table A-l of Appendix A 
(Volume 1) presents details of the analysis. The following 
subsections describe the factors and list the unrelated 
measures. 

4 . 2 . 1 . 1 Software Engineering Factors 

The five Software Engineering factors are described -below. 
Each description is composed of the factor's name, -the-^per- 
centage of the variance the factor explains, a list of meas- 
ures correlated with the factor, and a narrative. The 
measures correlated with each factor ire listed in the order 
in which they are listed in Section A.l of Appendix A (Vol- 
ume 2) , first for the positively correlated measures and 
then for the negatively correlated measures. Correlations 
appear in parentheses following the measure codes. 

1. Factor 1 - General Software Engineering Practices 
(51 percent of variance) . With the exception of 

a. MT09 (+0.06) — Use of top-down design procedures 

b. TS06 (+0.25) --Use of structured FORTRAN precompiler 

all measures in this class are at least moderately corre- 
lated (0.444) with this factor; most are strongly correlated 
(0.561) with it. 

The level of correlation is consistent with the application 
of software engineering technology in the flight dynamics 
area. In general, most methods, techniques, and practices 
are part of what has become, for some developers, a standard 
approach to software development. Depending on a particular 
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team's motivation, especially the motivation provided by its 
project manager and leader, the degree of application of the 
approach varies from a beginner's reluctant use to an expe- 
rienced advocate's aggressive use. 

The most strongly correlated (£ 0.91) measures with this 
factor include the total use of design methods (MT81) , cod- 
ing methods (MT82) , all methods (MT84) , tools (TS81) , docu- 
mentation procedures (DC81) , and overall software 
engineering (SE81) . However, the total use of testing meth- 
ods (MT83) has the lower correlation of 0.78 and points out 
a Known weakness in the development process of this environ- 
ment; that is, the testing approach is the least consistent 
aspect of the development process and is a major concern for 
development managers. 

2. Factor 2 - Batch Development (9 percent of variance) . 

The only measure strongly correlated with this factor is 

• TS10 (-0.61)--Use of terminals for interactive de- 
velopment. High use of terminals corresponds to 
low measure values. 

Measures moderately correlated with this factor include 

a. MT04 (+0.51) --Use of formal design review procedures 

b. TS 0 2 (+0.55) — Use of informal development training 

procedures 

c. TS06 (+0.52) — Use of structured FORTRAN precompiler 

d. DC04 (+0.54) — Use of semiformal quality assurance 

procedures for documentation 

e. MT20 (-0.49) --Use of code configuration control 
procedures 

This factor underscores the benefits of the electronics 
technology boom and progress in the development process; it 
also highlights an environmental deficiency, an archaic 


batch computing facility. It also emphasizes a development 
manager's problem resulting from the conflict of the envi- 
ronmental deficiency with the other two benefits. 

The interactive developer has emerged in. parallel with hard- 
ware and electronics technology gains. However, although 
the flight dynamics environment makes terminals available 
fbr development, in reality, it is primarily a batch envi- 
ronment with very unreliable computers (6- to 8-hour mean 
time to failure) . Therefore, development managers have con- 
tinually faced developers' increasing fascination for and 
dependence on the use of graphic terminals for development 
(TS10) . Because development managers have not been able to 
establish the rigorous discipline necessary to develop soft- 
ware interactively with unreliable computers, tea. 's that 
develop interactively tend to be less coordinated and to 
behave more like separate groups of individuals each with 
responsibility for a different piece of the system. Hence, 
it is difficult to control the configuration of the system's 
code (MT20) . 

This factor also shows improvements in the development proc- 
ess (MT04, TS02, TS06, and DC04) , which have emerged in par- 
allel with the interactive developer in this environment. 

3. Factor 3 - Top-Down Design (8 percent of variance) . The 
measures strongly correlated with this factor include 

a. MT09 (+0.85) — Use of top-down design procedures 

b. MT10 (+0.69) — Use of iterative enhancement design 
procedures 

c. MT26 (+0.61) — Use of batch testing procedures 

The projects in this sample are similar and provide design 
and/or implementation models for subsequent projects; as a 
result, top-down design is not used extensively. However, 
this factor shows that the projects that use top-down design 
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methodology tend to rely on batch testing procedures. .Fac- 
tor 30 in Section 4.2.6 confirms this with a different class 
oi? measures (Models class) . An alternative design technique t 
iterative enhancement (MT10 ) , also shows this effect. 

4. Factor 4 - Structured Implementation (7 percent of vari- 
ance ) . The only measure strongly correlated with this fac- 
tor is 

v \ 

e TS06 (+0.62) — Use of structured FORTRAN precompiler 

Two other measures that are moderately correlated are 

i ' y / I) 

a. MT17 (+0.54) --Use of structured coding techniques! 

b. TS09 (-0.53) —Development of data generators for 
testing 

This factor represents the use of a structured FORTRAN pre- 
compiler to aid in producing structured code. The inverse 
correlation with the development of data generators for 
testing is coincidental. Over the timespan of the projects 
in this sampler the use of a structured FORTRAN precompiler 
has increased from no use to total use. Over the same time- 
span f the availability of simulators and data support has 
increased significantly but not regularly; therefore/ the 
need for and use of data generators decreased/ but not nec- 
essarily proportionally. 

5. Factor 5 - Development Team's Organization (5 percent of 
variance) . None of the measures are strongly correlated 
with this factor. Measures that are moderately correlated 
include 

a. TS07 (+0.47) — Use of standard computer system code 
checking tools 

b. MT01 (-0.45) --Use of chief programmer; i.e. f proj- 
0 ect leader directs all development activity 


c. DC 08 (-0.46) -^Relevance of user's guide and system 
description . 

Jl. .. ; 

- •’ \ 

d. DC10 (-0.47) — Relevance of weekly/monthly progress 
reports 

In this environment, the project leader is the lead program- 
mer of the development team and is responsible for weekly/ 
monthly reporting and quality assuring the user's guide and 
system description. The inverse correlation of using com- 
puter system code checking tools (TS07) with the degree of 
using a lead programmer (MT01) seems to be due to the effect 
of the smaller projects, which are generally led by more 
junior personnel who are not as familiar with the computer 
system as more senior project leaders. 

4. 2.1.2 Measures Unrelated to Software Engineering 

The following list contains measures that are not correlated 
with any of the Software Engineering factors at the 0.01 
significance level. The factor with which each measure is 
most strongly correlated and the value of that correlation 
appear in parentheses after the measure code. 

1. TS02 (factor 2, +0.55) — Informal training in devel- 
opment 

2. TS09 (factor 1, +0.55) — Development of data genera- 
tors for testing 

3. DC09 (factor 1, +0.56) --Formal treatment (edit text 
and prepare artwork) of user's guide and system 
description 
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4.2.2 DEVELOPMENT TEAM ABILITY CLASS 

The factor analysis of the Development Team Ability class of 
measures produced a model containing six factors, which ex- 
plain 82 percent of the variance of the class. Together 
they describe the general level of the technical staff's 
ability and five other independent effects. Table A- 2 of 
Appendix A (Volume 1) presents details of the analysis. The 
following subsections describe the factors and list the un- 
related measures. 

4. 2.2.1 Development Team Ability Factors 

The six Development Team Ability factors are described 
below. Each description is composed of the factor's name, 
the percentage of the variance the factor explains, a list 
of measures correlated with the factor, and a narrative. 

The measures correlated with each factor are listed in the 
order in which they are listed in Section A. 2 of Appendix A 
(Volume 2) , first for the positively correlated measures and 
then for the negatively correlated measures. Correlations 
appear in parentheses following the measure codes. 

1. Factor 6 - Technical Staff's Ability (36 percent of var- 
iance) . The measures strongly correlated with this factor 
include 

a. AP03 (+0.75) --Contribution by expert 3 

b. AP08 (+0.57) — Application experience of programmers 

c. AP81 (+0.82) --Contribution by experts 

d. AP8x (> +0.73) — Overall application experience of 

team (x » 2 and 4) ; 

e. MGxx (> +0.82) — Management effectiveness of proj- 
ect leader (xx * 2, 8, 14, 20, and 26) 
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f» MGxx (>_ +0.61) — Overall management effectiveness 
o f manager* except development interface leader 
(xx * 81 through 85, 87, 88, and 93) 

g. PFxy (,>+0.65) — On-the-job performance of pro- 
grammers alone and technical staff (x * 0 through 
3; y * 1 though 4) 

h. ABxx (>_ +0.90) --Overall team ability (xx * 81 
through 92) 

This factor indicates the overall ability of the technical 
staff with emphasis on contributions from experts (AP81) , 
the project leader's effectiveness during implementation and 
testing (MG14, MG20, and MG26) , and the on-the-job perform- 
ance of the programmers weighted by the project manager and 
leader (PFx2) , i.e. , the basic team. 

2. Factor 7 - Development Interface Managers' Effectiveness 
and Performance (19 percent of variance) . The measures 
strongly correlate! with this factor include 

a. AP09 (+0.63) — Application experience of analysts 

b. MGxx (>, +0.56) —Management effectiveness of devel- 
opment interface manager and leader (xx * 5, 6, 11, 
12, 17, 18, 23, 24, 29, 30, 91, and 92) 

c. MGxx (_> +0.64) — Management effectiveness of proj- 
ect manager during testing and overall life of 
project (xx * 19, 25, and 87) 

d. MGxx (> +0.60) —Overall management effectiveness 
during implementation and testing (xx ■ 83 through 
85) 

e. PFx9 (>, +0. 64) -On-the-job performance of devel- 
opment interface manager (x * 0 through 3) 

f. MG03 (-0.66) — Management effectiveness of analysis 

during preliminary design 
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g. PFxy (<_ -0.56) --On-the-job performance of devel- 
opment project and analysis managers and analysis 
interface manager (x * 0 through 3; y »•' 5, 6, and 8) 

This factor represents the effectiveness and on-the-job per- 
formance of the development interface manager and the effec- 
tiveness of the development interface leader* This factor 
also shows a correlation with the analyst's application ex- 
perience (AP09) and an inverse correlation with the analysis 
manager's on-the-job performance (PFxy) . 

3. Factor 8 - Managers' Stability and Performance (11 per- 
cent of variance) . The measures strongly correlated with 
this factor include 

a* MG09 (+0.68) --Management effectiveness of analysis 
manager during detailed design 

b* MGxx (>,+0.72) — Management effectiveness of anal- 
ysis leader during testing (xx » 22 and 28) 

c. MGxx (>,+0.57) — Stability of development project 
leader and overall stability of managers (xx * 32, 
35, and 86) 

d. PFxy ( >, +0. 58) --On-the-job performance of devel- 
opment project and interface managers (x * 0 
through 3; y * 7 and 9) 

This factor represents both the on-the-job performance of 
the development team managers (i.e., the project manager and 
leader and the development interface manager and leader) and 
the stability of (few number of changes in) those posi- 
tions. There are also correlations with the analysis mana- 
ger's effectiveness during detailed design and the analysis 
leader's effectiveness during testing. 

In this environment, the analysis manager has considerable 
control over the definition of the functional specifications 
and requirements and changes to them. Most of the activity 
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in these areas occurs before the critical design review 
(CDR) . The analysis leader becomes a more prominent figure 
during implementation and is the key individual involved in 
preparing for and conducting acceptance testing. 

4. Factor 9 - Analysis Manager's Effectiveness (7 percent 
of variance) . The measures strongly correlated with this 
factor include 

a. MGxx (>_ +0.60) — Management effectiveness of anal- 
ysis manager during implementation and system test 
ing (xx » 15 and 21) 

b. MG89 (+0.74) — Overall effectiveness of analysis 
manager 

c. AP83 (-0.65) --Development team familiarity with 
project and teammates 

This factor represents the effectiveness of the analysis 
manager. It appears that the analysis manager is most ef- 
fective when members of the development team do not partici 
pate in functional specifications and requirements 
definition (AP10, -0.56) and do not work together much 
before the project (AP12, -0.45). AP83 is the sum of AP10 
through AP12. 

5. Factor 10 - Analysis Leader's Effectiveness (5 percent 
of variance) . The measures strongly correlated with this 
factor include 

a. MGxx (> +0.58) — Management effectiveness of anal- 
ysis leader during detailed design and implementa- 
tion (xx = 10 and 16) 

b. MG90 (+0. 59) --Overall effectiveness of analysis 
leader 

c. AP04 (-0. 66) --Contribution by expert 4 
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This factor is named for MG90, although the analysis 
leader's effectiveness during detailed design (MG10) is the 
most strongly correlated (+0.68) measure. The inverse cor- 
relation with contributions by expert 4 (AP04) is partially 
coincidental; that is, the higher levfil organization fre- 
quently staffs the development team with more senior person- 
nel when there is a potentially weak analysis team and vice 
versa. 

6. Factor 11 - Project Manager's Experience and Stability 
(5 percent of variance) . The only measure strongly corre- 
lated with this factor is 

e MG31 (-0. 69) —Stability of development project man- 
ager 

However, the project manager's experience with the applica- 
tion (AP06) is moderately correlated (+0.52) with the fac- 
tor. The inverse correlation with MG31 indicates that more 
experienced project managers are likely to be promoted to 
higher level management positions or moved to manage new 
projects. 

4. 2. 2. 2 Measures Unrelated to Development Team Ability 

The following list contains measures that are not correlated 
with any of the Development Team Ability factors at the 0.01 
significance level. The factor with which each measure is 
most strongly correlated and the value of that correlation 
appear in parentheses after the measure code* 

1. AP05 (factor 10, +0.55) — Contribution by expert 5 

2. AP06 (factor 11, +0.52) — Project manager's experi- 
ence 

3. AP07 (factor 6, +0.47) — Project leader's experience 

4. API 0 (factor 9, -0.56) — Development team part icipa- 
- tion in requirements definition 
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AP11 (factor 11, -0.35) — Development team partici- 
pation in design 

AP12 (factor 12, -0.45) — Number of development team 
interactions before start of project 

MG01 (factor 6, +0.50) — Effectiveness of project 
manager during preliminary design 

MG04 (factor 9, +0.47) — Effectiveness of analysis 
leader during preliminary design 

MG06(factor 1 , +0.50) — Effectiveness of development 
interface leader during preliminary design 

MG07 (factor 6, +0.55) — Effectiveness of project 
manager during detailed design 

MG13 (factor 6, +0.57) --Effectiveness of project 
manager during implementation 

MG27 (factor 8, -0.48) — Effectiveness of analysis 
manager during acceptance testing 

MG33 (factor 9, -0.39) — Stability of analysis man- 
ager 

MG 3 4 (factor 7, -0.53) — -Stability of analysis leader 

PP15 (factor 6, +0.55) -On-the-job performance of 
development project managers during implementation 
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4.2*3 DIFFICULTY OF PROJECT CLASS 


The factor analysis of the Difficulty of Project class of 
measures produced a model containing five factors , which 
explain 74 percent of the variance of the class. Together 
they describe the general level of project difficulty and 
four other independent effects. Table A-3 of Appendix A 
(Volume 1) presents details of the analysis. The following 
subsections describe the factors and list the unrelated 
measures. 

4 . 2 . 3 . 1 Difficulty of Project Factors 

The five Difficulty of Project factors are described below. 
Each description is composed of the factor's name, the per- 
centage of the variance the factor explains, a list of meas- 
ures correlated with the factor, and a narrative. The 
measures correlated with each factor are listed in the order 
in which they are listed in Section A. 3 of Appendix A (Vol- 
ume 2 ) , first for the positively correlated measures and 
then for the negatively correlated measures. Correlations 
appear in parentheses following the measure codes. 

1. Factor 12 - Project Difficulty (30 percent of vari- 
ance) . The measures strongly correlated with this factor 
include 

a. CPxx (>, +0.61) — Complexity of normal processing 
(xx * 3 through 5) 

b. CP07 (+0.93) — Number of subsystems 

V. / . . , 77: . ■ . / j . 

c. CPQ8 (+0.67) — Number of data sets 

d. CP10 (+0.60) --Number of new algorithms 

e. CPxx (> +0.59) — Overall complexity and totals of 
constraints, processing, and communications 

•• (xx * 81, 82, 83, and 85) 

■: ; f . INxx i > +0 * 58 ) — Overtime conditions (xx ■ 1 and 2) 
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g. IN06 (+0.73) — Staffing problems during testing 
phases 

h. IN11 (+0 . 81) — Development team with poor attitude 

i. IN13 (+0.58) — Project leader turnover 

j. INxx (£ +0. 60) "-Totals of overtime conditions , 
staffing problems, team leadership problems, and 
overall internal influences (xx ■ 81 through 84) 

k. EX01 (+0.76) — Changing requirements 

l. EX06 (+0.73) — Unresponsiveness of development in- 
terface leader 

m. EX07 (+0.68) — Number of subsystems developed by 
external development group 

n. EXxx (> +0. 63) --Overall poor quality and insta- 
bility of requirements and problems with external 
development group (xx * 81 and 83) 

o. DF81 (+0.92) — Overall difficulty of project, in- 
cluding complexity and effects of internal and ex- 
ternal influences 

This factor represents the overall difficulty of a project. 
The three most significant effects are the number of subsys 
terns to be developed (CP07, +0.93) , the development team's 
attitude (IN11, +0.81), and changing requirements (EX01, 
+0.76) . 

2. Factor 13 - External Support (18 percent of variance) . 
The measures strongly correlated with this factor include 

a. EX02 (+0. 65) --Incompleteness of requirements 

b. EXxx (>_ +0.75) — Lack of analysis support (xx * 3 
and 4) 

c. EXxx (>, +0 . 77) — Simulator unavailability and poor 
data support (xx * 10 and 12) 


d. EX13 (+0.67) --Unresponsiveness o£ analysis leader 

e. EX18 (+0.58) — Poor hardware support 

f. EXxx (£ +0. 61) --Total lack of support from exter- 
nal groups and poor simulator and system software/ 
hardware support (xx * 82, 84, 86, and 87) 

This factor represents the effects of poor external support, 
especially poor analysis and simulator support. 

3. Factor 14 - Analysis Leader's Responsiveness and Sche- 
dule (10 percent of variance) ... The factors strongly corre- 
lated with this factor include 

a. EX14 (+0. 74) --Analysis leader turnover 

b. EX85 (+0. 62) --Overall unresponsiveness associated 
with analysis leader 

c. CPU (-0.72) — Development schedule difficulty 

This factor demonstrates an environmental effect. When 
schedules are difficult (short) , the higher level organiza- 
tion will not encourage, suggest, or allow a change in anal- 
ysis leaders because there may not be enough time for an 
effective and efficient phase-in of a new analysis leader. 
Furthermore, analysis leaders are reliable senior personnel 
responsible for accepting the software; therefore, with a 
difficult schedule, their sense of urgency intensifies, they 
are more cooperative, and they are more responsive to devel- 
opment problems. 

4. Factor 15 - Analysis Leader 1 s General Responsiveness 
(8 percent of variance) . The measures strongly correlated 
with this factor include 

a. EX16 (+0. 62) --Unresponsiveness of analysis leader 
during late stages of development 

b. EX85 (+0.56) — Overall unresponsiveness associated 

with the analysis leader P 
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This factor also demonstrates an environmental effect. When 
there are few changes in analysis leadership (analysis man- 
ager and leader) . there is a higher probability that the 
analysis leader will be more responsive in the later stages 
of development. Combined with factor 14 . the development 
team gets its best analytical support when the schedule is 
difficult and When there are few changes in analysis leader- 
ship. 

5. Factor 16 - High-Level Development Support (8 percent of 
variance) . The measures strongly correlated with this fac- 
tor include 

a. EX05 (+0.59) — Unresponsiveness of development in- 
terface manager 

b. EX17 (+0.59) — System software support problems 

c. EX86 (+0.57) — Total system software and hardware 
support problems 

d. IN05 (-0.66) — Development team turnover 

This factor demonstrates a subtle environmental effect. 

When the development interface manager (who is the final 
authority for development direction, cost, and contact with 
customer, support, and contractor groups) is unresponsive to 
development problems and when there is poor support for sys- 
tem support software and hardware (which is controlled indi- 
rectly by the development interface manager) . there is 
little development team turnover. Apparently, when the de- 
velopment team is not supported at the highest technical 
level, members stay with the job until it is complete; i.e.. 
they do not desert their teammates when development condi- 
tions are difficult. However, it is not likely that anyone 
encourages the development interface manager to be unsup- 
portive. v. : 



4 . 2 . 3 . 2 Measures Unrelated to Difficulty of Projecc 

The following list contains measures that are not correlated 
with any of the Difficulty of Project factors at the 0.01 
significance level. The factor with which each measure is 
? most strongly correlated and the value of that correlation 

i appear in parentheses after the measure code. 

•i 1. CP01 (factor 12, +0.54) — Memory storage constraints 

^ 2. CP02 (factor 14, +0. 54) —Execution timing con- 

| straints 

. 1 

3. CP06 (factor 12, +0.28) —Number of system programs 

J that communicate 

i 

4. CP09 (factor 15, +0. 52) —Amount of old code used 

’ J ' , ■ ■ : 

1 5. CP84 (factor 12, +0.49) — Total of unrelated com- 

plexity measures 

I ... 

1 6. IN07 (factor 14, -0.55) --Need for extra help during 

development 

7. IN08 (factor 14, -0.52) — Unresponsiveness of proj- 

n ect manager during earlier phases of development 

> 

1 ' 

’’ 8. IN09 (factor 13, -0.54) — Project manager turnover 

j 9. IN10 (factor 12, +0.53) — Unresponsiveness of proj- 

ect manager during later phases of development 

j 10. EX11 (factor 13, +0. 53) — Incorrectness of simulator 

11. EX15 (factor 15, +0.55) — Unresponsiveness of analy- 
|| sis leader during later phases of development 


r- 1 •; 

D 
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4.2.4 PROCESS AND PRODUCT CHARACTERISTICS CLASS 

The £actor analysis o£ the Process and Product Characteris- 
tics class o£ measures produced a model containing five fac- 
tors, which explain 85 percent of the variance o£ the 
class. Together they describe the general level of process 
and product quality and £our other independent effects. 

Table A- 4 of Appendix A (Volume 1) presents details of the 
analysis. The following subsections describe the factors 
and list the unrelated measures. 

4.2. 4.1 Process and Product Characteristics Factors 

The five Process and Product Characteristics factors are 
described below. Each description is composed of the fac- 
tor's name, the percentage of the variance. the factor ex- 
plains, a list of measures correlated with the factor, and a 
narrative. The measures correlated with each factor are 
listed in the order in which they are listed in Section A. 4 
of Appendix A (Volume 2) , first for the positively corre- 
lated measures and then for the negatively correlated meas- 
ures. Correlations appear in parentheses following the 
measure codes. 

1 . Factor 17 - Process and Product Quality (43 percent of 
variance) . With the exception of 

a. PRO! (+0.42) — Project cost 

b. PRxx (> +0.27) --Size of modules (xx = 4 through 7) 

all Software Product (PR) category measures and all Product/ 
Process Performance (PP) category measures are strongly cor- 
related with this factor. The most strongly correlated 
single measure, i.e. , not a sum, is development planning and 
follow-through (PP08, +0*97). The implication, of course, 
is that higher quality development processes and products 
result directly from good planning and following through 
With" the plans. 


2. Factor 18 - Resource Availability (21 percent of vari- 
ance) « With the exception of 

a. RA01 (+0.13) --Availability of formal training for 
development 

b. RAO 6 (+0.39) — Availability of simulator support 

c. RAxx (< +0.49) — Availability of development sup- 
port personnel (xx * 16 and 17) 

all Resources Available (RA) category measures are strongly 
correlated with this factor. The two most strongly corre- 
lated (+0.95) measures are the availability of support docu 
mentation describing the development process (RA03) and the 
availability of instruction in the use of support software 
(RA04) . 

3- Factor 19 - Module Size (11 percent of variance) . The 
measures strongly correlated with this factor are 

• PRxx (< -0.55) — Size of modules (xx » 4 through 7 
and 81) 

This factor is simply a measure of the average size of mod- 
ules in the product , assuming that small modules are better 
(i.e. , easier to design, implement, test, and maintain) . 
Larger Values of the measures correspond to smaller module 
sizes. 


4 • Factor 20 - Support Software Support (6 percent of vari 
ance) . The measures strongly correlated with this factor 
include 

a. RAO 6 (+0.64) --Availability of simulator support 

b. RA82 (+0.81) — Availability of support for system 
support software 

This factor primarily represents the availability of person 
nel to maintain system support software and to provide 





instruction in its use. It also represents the availability 
of a data simulator. 

5. Factor 21 - Formal Training (4 percent of variance) . 

The only measure strongly correlated with this factor is 

• RA01 (-0. 58) --Availability of formal training for 
development 

The factor is self-explanatory; however , the availability of 
librarians (RA16) is moderately positively correlated 
(-1*0 .52) . Over the timespan of this sample, librarians have 
always been available for a fairly high level of support. 
Unfortunately, because of expense and time constraints, for- 
mal training in development is very limited. 

4,2. 4. 2 Measures Unrelated to Process and Product Character - 
istics 

The following list contains measures that are not correlated 
with any of the Process and Product Characteristics factors 
at the 0.01 significance level. The factor with which each 
measure is most strongly correlated and the value of that 
correlation appear in parentheses after the measure code. 

1. RA16 (factor 21, +0. 52) —Availability of librarians 

2. RA17 (factor 18, +0.49) — Availability of dedicated 

experts for help 

3. PR01 (factor 21, -0.43) — Cost of project 

4. PRO 4 (factor 19, -0.55)— Size of newly developed 

modules 


0 


4.2.5 DEVELOPMENT TEAM BACKGROUND CLASS 

The factor analysis of the Development Team Background class 
of measures produced a model containing five factors, which 
explain 86 percent of the variance of the class. Together 
they describe the level of the technical staff's applicable 
experience and reputation and four other independent ef- 
fects. Table A- 5 of Appendix A (Volume 1) presents details 
of the analysis. The following subsections describe the 
factors and list the unrelated measures. 

4.2. 5.1 Development Team Background Factors 

The five Development Team Background factors are described 
below. Each description is composed of the factor's name, 
the percentage of the variance the factor explains, a list 
of the measures correlated with the factor, and a narra- 
tive. The measures correlated with each factor are listed 
in the order in which they are listed in Section A. 5 of Ap- 
pendix A (Volume 2) , first for the positively correlated 
measures and then for the negatively correlated measures. 
Correlations appear in parentheses following the measure 
codes. 

1. Factor 22 - Technical Staff's Applicable Experience and 
Reputation (41 percent of variance) . The measures strongly 
correlated with this factor include 

a. RKxy (>_ +0.56) — In general, the reputation of 
programmers, technical staff, and project managers 
alone and combined with analysis managers (x = 0 
through 3; y = 1 through 6). (The sign of the cor- 
relation is reversed relative to that reported in 
Appendix A of this volume because low values corre- 
spond to high ranks.) 

b. YPxy (5^ +0.62) --In general, the professional ex- 
perience of programmers and development 


organization technical staff except for testing 
(x ■ 0, 1, and 3; y ■ 1, 2, 3 , and 6) 

c. YAxy (£ 40.60) — In general, the applicable ex- 
perience of programmers, technical staff, and man- 
agers, excluding development inter face -manager s 

(x =» 0 through 3; y ■ 1 through 6) 

d. YEXy (> +0.63) — In general, the environment ex- 
perience of programmers, development organization 
technical staff, and project managers (x • 0 
through 3; y * 1, 2, 3, and 5) 

e. ZZx9 (< -0,71) — Experience of development inter- 
face managers (ZZ * RK, YP, YA, and YE; x ■ 0 
through 3) 

This factor basically represents the technical staff's ap- 
plicable experience and reputation (rank) , where the techni- 
cal staff includes the programmers (y = 1 through 4) and 
some fraction of the project managers, (y * 2 through 7), 
the analysis managers (y * 3, 6, and 8), and the development 
interface managers (y * 4, 7, and 9). The factor shows that 
reputation is correlated with type of experience in the fol- 
lowing order: applicable, environment, and professional. 

The inverse correlation with the development interface man- 
agers results because their organization is more stable; 
therefore, they tend to have more experience than the devel- 
opment organization for the projects in the sample. 

2. Factor 23 - Technical Staff's Professional Experience 
(22 percent of variance) . The measures strongly correlated 
with this factor include 

a. YPxy (£ 4-0.57) --Professional experience of pro- 
grammers, technical staff, and project and develop- 
ment interface managers (x * 0 through 3; y * 1, 2, 
3 , 4 , 5, and 7) 
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b. YAx7 (£ +0.73) — Applicable experience of project 
and development interface managers (x * 0 through 3) 

c. XAx4 (£ +0.61) — Applicable experience of program- 
mers and development managers for testing and over- 
all life of the project (x * 2 and 3) 

d. RK06 (-0. 57) --Reputation of project and analysis 
managers for design 

e. RK28 (-0.60)' — Reputation of analysis managers for 
testing 

f . YAx8 ( <, -0.58) — Applicable experience of analysis 
managers (x « 0 through 3) 

g. YE18 (-0. 58) —Environment experience of analysis 
managers for implementation 

Basically, this factor represents the technical staff's pro- 
fessional experience. It also shows inverse correlations 
with the analysis manager's experience (especially applica- 
ble experience) for different phases of development. 

3. Factor 24 - Development Interface Managers' Environment 
Experience (10 percent of variance) . The measures strongly 
correlated with this factor include 

a. RKx7 (> +0.57) — Reputation of development inter- 
face managers (x * 0, 2, and 3) 

b. YExx (£+0.56) — Environment experience of devel- 
opment interface managers alone and combined with 
programmers (xx = 7, 14, 17, 24, 27, 34, and 37) 

This factor represents the environment experience of the 
development interface managers. The most strongly corre- 
lated (+0.71) measure is YE14, which combines their experi- 
ence with the basic development team's experience, 


4. Factor 25 - Project and Analysis Managers' Environment 
Experience for Implementation (7 percent of variance) . The 
measures strongly correlated with this factor include 

a. YA05 (+0. 58) --Applicable experience of project man- 
agers for design 

b. YE16 (+0.59) — Environment experience of project and 
analysis managers for implementation 

c. YEx8 (> +0.57) — Environment experience of analy- 
sis managers overall and for testing (x - 2 and 3) 

This factor is named for the most strongly correlated meas- 
urer YE16. 

5. Factor 26 - Analysis Managers' Environment Experience 
for Testing (6 percent of variance) . The measures strongly 
correlated with this factor include 

a. YE26 (+0. 59) --Environment experience of project and 
analysis managers for testing 

b. YE 2 8 (+0.65) — Environment experience of analysis 
managers for testing 

c. RK17 (-0. 62) --Reputation of development project and 
interface managers for implementation 

This factor is named for the most strongly correlated meas- 
ure, YE28 . 

4. 2, 5. 2 Measures Unrelated to Development Team Background 

The following list contains measures that are not correlated 
with any of the Development Team Background factors at the 
0.01 significance level. The factor with which each measure 
is most strongly correlated and the value of that correla- 
tion appear in parentheses after the measure code. 

1. RK08 (factor 23, -0.45) — Reputation of analysis 
managers during design < 


m mmm 





jj 


RK15 (factor 26, -0. 54) --Reputation of development 
project managers during implementation 

YPx8 (factor 23, j> +0.45) — Professional years of 
experience of analysis managers during all phases 
(x * 0 through 3) 

YE08 (factor 23, +0.56) — Environment years of ex- 
perience of analysis managers during design 
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4.2.6 MODELS CLASS 


The factor analysis of the Models class of measures for re- 
source estimation models produced a model containing six 
factors, which explain 73 percent of the variance of the 
class. Together they describe the system's size and five 
other independent effects. Table A-6 of Appendix A (Vol- 
ume 1) presents details of the analysis. The following sub- 
sections describe the factors and list the unrelated 
measures. 

4. 2. 6.1 Models Factors 

The six Models factors are described below. Each descrip- 
tion is composed of the factor's name, the percentage of the 
Variance the factor explains, a list of the measures corre- 
lated with the factor, and a narrative. The measures corre- 
lated with each factor ate listed in the order in which they 
are listed in Section A. 6 of Appendix A (Volume 2) , first 
for the positively correlated measures and then for the neg- 
atively correlated measures. Correlations appear in paren- 
theses following the measure codes. 

1. Factor 27 - System Size (24 percent of variance) . The 
measures strongly correlated with this factor include 

a. WFxx (_> +0.66) — Overall complexity and size- 
related complexity measures (xx » 14, 15, 16, 22, 
23, and 85) 

b. WFxx (+0.87) --Total effort (xx = 51 and 52) 

c. WFxx (>_ +0.84) — Delivered and developed graphics 
macros, FORTRAN, and total source code (xx = 68 
through 70 and 72 through 74) . (Developed assem- 
bler code (WF67) has a correlation coefficient of 
+ 0 . 66 .) 

d. WF75 (+0.68) --Number of data base items 


e. WF76 (+0.93) --Number of pages of documentation 

f. PS09 (+0.72) —Schedule length 

g. WF44 (-0,62) — Use of chief programmer concept 

h. PS01 (-0.62) — Percentage of calendar schedule spent 
to get to CDR 

This factor represents the system's size. It shows that the 
larger a system becomes, the less likely it is that a single 
person (WF44) can direct all development activities. It 
also shows that the larger the system, the greater the pro- 
portion of calendar schedule needed to implement and test 
the system (PS01) . The number of pages of documentation 
(WF76) is the most strongly correlated (+0.93) measure. 

2. Factor 28 - Programmers' Qualifications (14 percent of 
variance) . The measures strongly correlated with this fac- 
tor include 

a. WFxx (> +0.59) — Programmer qualifications and 
graphics, application, and overall experience 
(xx = 4, 7, 8, and 81) 

b. WF54 (+0.65) — Percentage of work schedule needed to 
produce accepted software 

c. WFxx (_> +0. 65) --Percentage of code that is mathe- 
matical or computational (xx * 61 and 62) 

d. PSxx (> +0.72) — Percentage of calendar schedule 
spent in design and implementation rather than in 
testing only and wrapping up project (xx = 2 and 8) 

e . PSxx (_> +0.62) — Reduced complexity of project 
because of programmer experience (xx ■ 10, 11, and 
81) 

f. WF12 (-0.57) — Complexity of customer interface 
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This factor represents the programmers* overall qualifica- 
tions, it shows that more experienced programmers (1) spend 
little time on the project after the system is accepted 
(WF54) , (2) spend less time in the testing phases (PS02 and 
£308) , and (3) tend to work on systems that are more mathe- 
matical and computational (WF62) rather than on systems that 
are nonmathematical and require input/output (I/O) format- 
ting (WF61) . It may be coincidental, but the factor also 
shows that more experienced programmers tend to have a less 
complicated customer interface (WF12) . 

3. Factor 29 - Testing Strategy (13 percent of variance) . 
The measures strongly correlated with this factor include 

a. WF02 (+0.66) --Participation in requirements defini- 
tion 

b. WFxx (>_ +0.67) — Large amounts of graphics code 
and graphics testing (xx * 19, 31, 32, 36, 37, and 
66 ) 

c. WF63 (+0.57) — Percentage of code needed for central 
processing unit (CPU) and (I/O) control 

d. WF47 (+0. 69) --Percentage of programmer effort 

e. WF46 (-0.78) —Percentage of administrative manage- 
ment effort 

This factor identifies a testing strategy (correlated with 
code type) and possibly a management deficiency. Most of 
the software developed in this sample was developed for in- 
teractive operation, although it must also be possible to 
execute the software through batch operation. One method of 
testing the software is to use a graphic device as in normal 
operational use; a certain amount of testing must be done in 
this manner. However, graphics test time is difficult to 
obtain during standard hours. Furthermore, development 
groups are given lowest priority during standard hours. 


Therefore, more experienced managers and senior personnel 
try to discourage a strong dependence on graphics test time 
because programmers must work nonstandard hours to get effi- 
cient test time. r ) 

The possible management deficiency is that when programmers 
appear to be hard at work (graphics testing) , managers tend 
to think that everything is all right and to pay less atten- 
tion to the project (WP46) . In general, however, systems 
tested primarily through graphics testing do not have better 
operational performance records than those tested with more 
emphasis on batch methods. 

4. Factor 30 - New Project Type (10 percent of variance) . 

The measures strongly correlated with this factor include 

a. WF39 (+0.59) — Percentage of batch testing 

b. WF43 (+0.70) --Percentage of top-down development 

c. PS11 (+0.62) — Reduced complexity of project because 
of programmer experience 

d. PS14 (+0.65) --Percentage of new design 

e. PS15 (+0 .68) --Percentage of new code 

This factor represents the new project type, i.e. , mostly new 
design (PS14) and new code (PS15) . It shows that development 
teams developing systems with less of a design/ implementation 
model (1) tend to use top-down development techniques (WF43) , 

(2) tend to rely on batch testing techniques (WF39) , and 

(3) are composed of more experienced personnel (PS11) . In 
addition, see factor 3 in Section 4.2.1. 

5. Factor 31 - Code Reading and Testing (6 percent of vari- 
ance) . None of the measures are strongly correlated with 
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this factor* Measures that are moderately correlated with 
this factor include 

a. WP42 (+0.52) — Amount of code reading 

b. PSQ7 (+0.55) --'Percentage of calendar schedule spent 
during documentation phase 

c. PS05 (-0 .45) —Percentage of calendar schedule spent 
during testing phase 

d. PS06 (-0.52) --Percentage of calendar schedule spent 
during testing activity 

This factor shows that teams that rely on code reading 
(WF42) as a code validation technique spend less time in the 
testing phases (PS05 and PS06) . 

6 . Factor 32 - Design Team Size (6 percent of variance) . 
None of the measures are strongly correlated with this 
factor. Measures that are moderately correlated with this 
factor include 

a. WF05 (+0 .47) --Programmer familiarity with develop- 
ment computer 

b. WFxx (_> 0.45) — Complexity of interprogram commu- 
nications, data base structures, and execution tim- 
ing constraints (xx = 16, 18, and 21) 

c. WF03 (-0.48) — ’Percentage of programmers participat- 
ing in design 

d. WF50 (-0 .47) --Percentage of support service support 

This factor shows that projects with relatively small design 
teams (WF03) are composed of programmers who are experienced 
with the development computer (WF05) and who have to design 
a system with complex interprogram communications (WF16) , 
data base structures (WF18) , and execution timing con- 
straints (WF21) . In general, the use of a smaller design 
team is a development philosophy that has evolved over the 



past few years in this environment; that is, it is desirable 
to start projects with smaller, more experienced staffs to 
get a better grasp of the problem before plunging ahead* 

4.2. 6.2 Measures Unrelated to Models 

The following list contains measures that are not correlated 
with any of the Models factors at the 0,01 significance 
level. The factor with which each measure is most strongly 
correlated and the value of that correlation appear in pa* 
rentheses after the measure code. 

1. WP03 (factor 32, -0.49) — Percentage of programmers 
participating in design 

2. WF05 (factor 28, +0.54) — Programmer familiarity 
with development computer 

3. WFQ6 (factor 30, +0.39) — Programmer familiarity 
with programming language 

4. WF09 (factor 32, +0.40) — Degree to which program- 
mers worked together before project 

5. WF13 (factor 30, +0.49) — Number of customer- 
originated design changes 

6. WF18 (factor 27, +0. 55) —Complexity of data base 
structures 

7. WF2C (factor 27, +0. 55) --Memory storage constraints 

8. WF21 (factor 27, +0.46) — Execution timing con- 
straints 

9. WF34 (factor 32, -0.48) — Percentage of programmers 
participating in design 

10. WF35 (factor 28, +0.49) --Percentage of programmers 
who worked together before project 

11. WF40 (factor 29, -0.47) — Percentage of development 
that is interactive (TSO) ! 
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WF41 (factor 32, +0.37) — Percentage of code that is 
structured 


13. WF42 (factor 31* +0.52) — Percentage of code for 
which code reading is performed 

14. WF45 (factor 29, -0. 46) --Percentage of effort that 
is technical management 

15. WF50 (factor 30, +0.52) --Percentage of effort that 
is support services charges 

16. WF71 (factor 27, +0.54) — Number of delivered lines 
of code (LOC) that is assembler language 

17. PS03 (factor 27, +0. 50) --Percentage of calendar 
schedule spent during implementation phase 

18. PS04 (factor 27, +0.53) — Percentage of calendar 
Schedule spent during implementation activity 

19. PS05 (factor 28, -0.50) — Percentage of calendar 
schedule spent during testing phase 

20. PS06 (factor 27, +0.53) --Percentage of calendar 
schedule spent during testing activity 

21. PS07 (factor 31, +0.55) — Percentage of calendar 
schedule spent during documentation phase 

22. PS 12 (factor 28, -0.55) — Complexity of product 

23. PS 13 (factor 30, +0. 51) --Complexity of external 
development effects 

24. PS 16 (factor 30, +0.54) — Percentage of new testing 


4.2.7 ADDITIONAL DETAIL CLASS 

The factor analysis of the Additional Detail class of meas- 
ures produced a model containing six factors, which explain 
83 percent of the variance of the class. Together they de- 
scribe the system's size in terms of effort and five other 
independent effects. Table A- 7 of Appendix A (Volume 1) 
presents, details of the analysis. The following subsections 
describe the factors and list the unrelated measures. 

4. 2. 7.1 Additional Detail Factors 

The six Additional Detail factors are described below. Each 
description is composed of the factor's name, the percentage 
of the variance the factor explains, a list of the measures 
correlated with the factor, and a narrative. The measures 
correlated with each factor are listed in the order in which 
they are listed in Section A. 7 of Appendix A (Volume 2) , 
first for the positively correlated measures and then for 
the negatively correlated measures. Correlations appear in 
parentheses following the measure codes. 

1, Factor 33 - Effort-Related System Size (30 percent of 
variance) . The measures strongly correlated with this fac- 
tor include 

a. MSxx (> +0.60) — Number of subsystems, input data 
sets, and total data sets in product; number of 
programs, subsystems, and total data sets needed in 
normal processing (xx * 2, 3, 6, 11, 12, and 16) 

b. MXxx (_> +0 .70) --Pages of documentation (xx * 21 
through 25) 

c. MSxx ( _> +0.87) — Average staff size (xx = 26 
through 28) 

d. SWXy (> +0.57) — Basically, all representations 
that are proportional to LOC, i.e. , components, 
modules, executable LOC, decisions, library 


changes, software changes, and errors for newly de- 
veloped, extensively modified, slightly modified, 
and total representations. Usually, old, assem- 
bler , and graphics macros representations are not 
strongly correlated, (in general, x * 0, 2, 4, 5, 
6; y M or 9) 

e. SW84 (+0. 56) --Decisions per module with executable 
code 

f. SW87 (+0.62) — Executable LOC per baseline diagram 
component 

g* ESxx (£ +0.68) --All size, effort, and process 

activity measures except use of tertiary develop- 
ment computer (ES19) (xx * 1 through 18) 

This factor represents the activities related to major ef- 
fort areas. Measures that generally represent minor effort 
areas, e.g., the use of unchanged (old) code, and measures 
that represent smaller efforts, e.g., the use of assembler 
code (on the average, less than 2 percent of the product), 
have the weakest correlations. 

2 . Factor 34 - Data Base Size (10 percent of variance) . 

The measures strongly correlated with this factor include 

a. MS20 (+0.60) --Total data base size 

b. SWxx (<, -0.58) — Slightly modified and old LOC and 
executable LOC of graphics macros; old LOC and exe- 
cutable LOC of FORTRAN (Xx * 18, 19, 24, 38, 39, 
and 44) 

In general, the input and output data base size requirements 
do not vary significantly from project to project in this 
environment. Therefore, when a project has unusually large 
data base requirements (MS20) , the likelihood is small that 
the project will have many similarities to other projects. 
Hence, fewer old graphics macros and less old FORTRAN code 
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can be caused (an inverse correlation with SWxx) , The im- 
plication is that projects with large data bases probably 
require more e££ort to develop than the average project. 
Factor 30 (new project type) in Section 4.2.6 is moderately 
correlated (+0.47) with number o£ items in the data base 
(WF23) . 

3 . Factor 35 - Internal Data Set Communication (6 percent 
of variance) . The measures strongly correlated with this 
£actor include 

a. SWxx (_> +0.58) — Old and total assembler LOG and 
executable LOC (xx * 14, 15, 34, and 35) 

b. MSxx ( <_ -0.58) — Number of I/O data sets in prod- 
uct and number needed for normal processing; number 
of I/O data base items needed for normal processing 
(xx * 8, 14, and 18) 

c. SWxx (£ -0.56) — Extensively modified baseline 
diagram components, decision modules, assembler 
LOG, and executable LOC (xx * 2, 7, 12, and 32) 

This factor represents the data processing system, i.e. , 
systems that preprocess data (e.g., for screening or cali- 
bration) or systems that postprocess data (e.g., for smooth- 
ing, quality assurance, or packaging) . These systems tend 
to use larger numbers of internal data sets to manipulate 
data. The factor also indicates that data processing sys- 
tems tend to use large amounts of extensively modified code 
and small amounts of assembler code. 

4. Factor 36 - Use of Old Code (6 percent of variance) . 

The measures strongly correlated with this factor include 

a. SW09 (+0.65) — Number of old modules 

b. SWxl (< -0.57) — New assembler LOC and executable 
LOC (x * 1 and 3) 
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This factor represents the use of old code* It indicates 
that as more old code is used, the less likely it is that 
new assembler code will be developed. Several other old 
code type measures are moderately positively correlated. 

5. Factor 37 - Code Error Content (5 percent of variance) . 
The measures strongly correlated with this factor include 

a. SW76 (-0.58) --Errors per 1000 LOC during development 

b. SW79 (-0.57) — Errors per baseline diagram component 

during development 

This factor represents the number of errors in the code. 
Several other error measures are moderately negatively cor- 
related. 

6 . Factor 38 * Code Complexity (4 percent of variance) . 

The measures strongly correlated with this factor include 

a. MSI 5 (+0. 59) --Number of output data sets 

b. SW82 (+0.67) — Decisions per 1000 executable LOC 

c. SW86 ( -0. 65) --Executable LOC per 1000 LOC 

d. SW88 (-0.59) —Executable LOC per module 

This factor represents the complexity of the code as meas- 
ured by decisions, i.e., number of simple arguments in IF, 
IF-THEN-ELSE , DO, and DOWHILE statements. It shows that as 
the number of decisions per thousand executable LOC (SW82) 
increases, the number of executable LOG per thousand LOC 
(SW86) and per module (SW88) decreases. 

4 . 2.7. 2 Measures Unrelated to Additional Detail 

The following list contains measures that are not correlated 
with any of the Additional Detail factors at the 0.01 signi- 
ficance level. The factor with which each measure is most 
strongly correlated and the value of that correlation appear 
in parentheses after the measure code. 


.... 
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1. MS 01 (factor 35, +0. 53) --Number of programs in de- 
veloped product 

2. MS04 (factor 35, «0, 56)— Number of I/O data sets in 
developed product 

3. MS 05 (factor 33, +0. 47) -"-Number of output data sets 
in developed product 

4. MS07 (factor 33, +0.46) — Number of input data base 
items in developed product 

5. MS09 (factor 35, +0.42) —Number of output data base 
items in developed product 

6. MS10 (factor 34, +0.56) — Total number of data base 
items in developed product 

7. MS13 (factor 33, +0.46) — Number of input data 3ets 
needed for normal processing 

8. MS17 (factor 34, +0.28) — Number of input data base 
items needed for normal processing 

9. MS19 (factor 35, +0. 43) --Number of output data base 
items needed for normal processing 

10. SW04 (factor 36, +0* 56) --Number of unchanged (old) 
baseline diagram components in product 

11. SW17 (factor 33, +0. 49) --Number of extensively mod- 
ified LOC of graphics macros 

12. SW29 (factor 36, +0. 55) --Total number of old LOC 

13. SW33 (factor 33, +0.55) — Number of slightly modi- 
fied executable LOC of assembler code 

14. SW37 (factor 33, +0.50) — Number of extensively mod- 
ified executable LOC of graphics macros 

15. SW49 (factor 34, -0.54) --Total number of old execu- 
table LOC 
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. SW69 (factor 33, +0.51) — Number of software errors 
in old code 


17. 

SW71 

(factor 

38, 

+0.37) — Percentage 

Of 

comments 

in 


newly 

developed 

code 




18. • 

SW72 

(factor 

33, 

+0. 40) — Percentage 

of 

comments 

in 


extensively i 

modified code 




19. 

SW73 

(factor 

33, 

+0.49) — Percentage 

of 

comments 

in 


slightly modified code 




20. 

SW74 

(factor 

33, 

-0. 21) — Percentage 

of 

comments 

in 


old code 


„ 




21. 

SW75 

(factor 

33, 

+0. 31) --Percentage 

of 

comments 

in 


all code 






22. 

SW77 

(factor 

37, 

-0. 51) — Errors per 

1000 executable 


LOC 







23. 

SW78 

(factor 

38, 

+0. 44) — Errors per 

1000 decisions 

24. 

SW80 

(factor 

37, 

-0.56) — Errors per 

decision module 

25. 

SW81 

(factor 

34, 

-0.50) — Decisions per 

1000 LOC 


26. 

SW83 

(factor 

33, 

+0.49) — Decisions per 

baseline 



diagram component 




27. 

SW85 

(factor 

38, 

-0.42) — Ratio of coded LOC to ex- 


panded LOC, i.e. , expansion from INCLUDE blocks of 
code 

28. SW89 (factor 38, +0.33) — Number of library data 
sets changed per implementation change 

29. SW90 (factor 35, +0.38) — Percentage of errors in 
implementation changes 

30. ESI 9 (factor 33, -0.15) — Number of computer hours 
used on tertiary development computer 
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4.3 SUMMARY ANALYSIS OF CLASS FACTORS 


The analyses described in Section 4.2 identify 38 principal 
factors among the 7 classes of measures; this number of fac- 
tors is substantially fewer than the 608 measures originally 
used in the analyses. The information contained in each set 
of measures (38 and 608) can be displayed for c parison 
using the cluster analysis technique discussed in Sec- 
tion 3.3. 

Figure 4-1 shows the results of a cluster analysis of the 
608 measures that passed the test of normality (see Sec- 
tion 4.1). Levels of similarity between projects are indi- 
cated in the table by the number of horizontal lines of as- 
terisks connecting them. For example. Figure 4-1 can be 
used to divide the sample of 20 systems into 2 groups, or 
clusters, by tracing across the graph at the level where the 
"number of clusters" is equal to 2. The 20 systems appear 
to be divided into these 2 clusters on the basis of size. 
This result verifies our intuitive understanding that essen- 
tial differences exist between small and large projects. 

System 730, which was grouped by cluster analysis with the 
small systems, is the only exception to the classification 
of systems with fewer than 30,000 lines of code as being 
small (see Section 2.1). Thus, System 730 is closer to the 
pattern of small systems than to that of large systems. 
Furthermore, it is the smallest of the large systems (with 
33,000 line?? of code). This suggests that the classifica- 
tion criterion should be adjusted to include this system in 
the small group. 

The importance of size measures is reflected in the results 
of the factor analyses: size factors are identified within 

three of the seven classes of measures. However, one prop- 
erty of factors is that they combine the effects of related 
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Figure 4-1. Cluster Map Based on Screened Measures 
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measures. Thus, the relative weights of the most frequently 
occurring measures, such as size, are reduced. 

Figure 4-2 shows the result of a cluster analysis of the 
38 factors identified in Section 4.2. Note that the classi- 
fication of projects into two groups on the basis of size 
has been preserved, although the structure within these 
groups as shown in Figure 4-2 is different from that of Fig- 
ure 4-1. The preservation of this basic classification con- 
firms that the factor analyses of classes of measures have 
been successful in extracting the major descriptive quali- 
ties from the original screened measures. 
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4 . 4 FACTOR ANALYSIS OF CLASS FACTORS 


The factor analysis of the combined (class) factors data set 
produced a model containing 5 high-level factors, which ex- 
plain 64 percent of the variance of the 36 class factors. 
Together they describe the small and large systems and the 
experience levels of the development teams. The details of 
this factor analysis are reproduced in Table B-l of Appen- 
dix B. The following subsections summarize these results. 
The high-level factors and the unrelated class factors are 
listed. 

4.4.1 HIGH-LEVEL FACTORS 

The five high-level factors are described below. Each de- 
scription is composed of the high-level factor's name, the 
percentage of the variance the high-level factor explains, a 
list of class factors correlated with the factor, and a nar- 
rative. The class factors correlated with each factor are 
listed in numerical order, first for the positively corre- 
lated class factors and then for the negatively correlated 
class factors. Correlations appear in parentheses following 
the class factor number. 

The descriptive names applied to factors in this section are 
intended to suggest the types of values that tend to occur 
together, based on the analysis of the factor pattern shown 
in Table B-lb of Appendix B. For example, two groups of at- 
tributes can be defined that are characteristic of small 
systems and of large systems. Depending on its size, any 
given system would possess all these attributes to a greater 
or lesser degree. Therefore, this section describes a set 
of profiles or patterns with which any given development 
project can be compared. 
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8623 


of variance) . The factors strongly correlated with this 
high-level factor include 

a. Factor 8 (+0 .60) —Stability and high performance of 
development team managers, i.e., project manager 
and leader and development interface manager and 
leader (MG and PF) 

b. Factor 13 (+0.67) — Poor external support (EX) 

c. Factor 17 (+0.79) — High-quality process and product 
(PR and PP) 

d. Factor 12 (-0.62) — Less complex and difficult proj- 
ect (CP, IN, EX, and DF) 

e. Factor 26 (-0.63) — Low level of environment experi- 
ence of analysis managers for testing (RK and YE) 

f. Factor 27 (-0.60) — Small system size (WF and PS) 

g. Factor 33 (-0.61) — Low effort-related system size 
(MS, SW, and ES) 

Factors that are moderately correlated include 

a. Factor 1 (+0.52) — Moderately high general use of 
software engineering practices (MT, TS, DC, and SE) 

b. Factor 5 (+0.54) — Moderately high use of chief pro- 
grammer. The sign of the correlation is reversed 
because the signs of the correlations of the prin- 
cipal measures are negative (MT, TS, and DC) 

c. Factor 6 (+0.50) — Moderate level of ability of 
technical staff (AP, MG, PF, and AB) 

d. Factor 29 (+0,47) — Moderate tendency to use 
graphics test time rather than batch testing meth- 
ods (WF) 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

D 

1 





e. Factor 35 (+0.48) — Moderately large number of data 
sets for internal communication. The sign of the 
correlation is reversed because the signs of the 
correlations of the principal measures are negative 
(MS and SW) 

f. Factor 37 (-0.49) --Smaller number of errors in the 
code during development. The sign of the correla- 
tion is reversed because the signs of the correla- 
tions of the principal measures are negative (SW) 

This high-level factor represents the typical small system 
development pattern. In this environment, small systems 
generally 

• Tend to be mainly utility data processing applica- 
tions, such as preprocessing for data reduction and calibra- 
tion or postprocessing for data smoothing, quality 
assurance, and packaging. Therefore, small systems use more 
internal data sets for communication (factor 35) to manipu- 
late data. Because small systems are usually data manipula- 
tion and inspection systems, they tend to have a large 
amount of interactive graphic code and, therefore, are pri- 
marily tested graphically (factor 29) . 

• Are not complex or difficult (factor 12) . They are 
developed with a high-quality process that produces a high- 
quality product (factor 17), which has moderately fewer de- 
velopment errors per thousand LOC (factor 37). 

• Tend to have chief programmers (factor 5) because 
the systems are small enough for a single person to direct 
all technical activity. 

• Have moderately able technical staffs (factor 6), 
who use software engineering practices to a moderately high 
degree (factor 1) . 


• Have more stable team management (factor 8) because 
the development schedules are relatively short. 

• Have poor external support (factor 13) because 
small systems are generally not so critical in terms of cost 
and schedule impact compared with larger projects being de- 
veloped at the same time. This is reenforced by the low 
level of environment experience of the analysis managers 
assigned responsibility for small systems during the testing 
phases (factor 26) . 

2 . High-Level Factor 2 - Large System Profile (14 percent 
of variance) . The factors strongly correlated with this 
high-level factor include 

a. Factor 31 (+0.58) --High use of code reading and 
less time spent during testing phases (WF and PS) 

b. Factor 33 (+0.64) — Large effort-related system size 
(MS, SW, and ES) 

c. Factor 2 (-0.67) — High use of interactive develop- 
ment techniques (MT, TS, and DC) 

d. Factor 4 (-0.59)--Low use of structured implementa- 
tion (MT and TS) 

e • Factor 7 (-0.64) — Ineffectiveness and low perform- 
ance of development interface managers (AP , MG, and 
PF) 

f. Factor 14 (-0 .60) --Difficult schedule and respon- 
sive analysis leader (CP and EX) 

Factors that are moderately correlated include 

a. Factor 12 (+0.54) — Moderately complex and difficult 
project (CP, IN, EX, and DF) 

b. Factor 27 (+0.47) --Moderately large system size (WF 
and PS) 
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c. Factor 10 (-0 .54) —Moderate overall ineffectiveness 
of analysis leader as a manager (AP and MG) 

d. Factor 18 (-0 .56) --Moderately low availability of 
resources (RA) 

e. Factor 19 (-0.44) — Large average module size. The 
sign of the correlation is reversed because the 
signs of the correlations of the principal measures 
are negative (PR) 

f. Factor 23 (-0.45) — Moderately low level of profes- 
sional experience of technical staff (RK, YP , YA, 
and YE) 

This high-level factor represents the typical large system 
development pattern. In this environment, large systems 
generally 

• Are moderately complex and difficult (factor 12) . 

• Have schedules with more than average difficulty; 
therefore, the analysis leader's responsiveness tends to be 
higher for larger projects (factor 14) . 

• Have technical staffs with a moderately low level 
of professional experience (factor 23). However, they tend 
to use primarily interactive development techniques (fac- 
tor 2) and code reading to validate code (factor 31) . Be- 
cause larger systems tend to reuse more code from previously 
developed systems than do smaller systems, structured imple- 
mentation is used less (factor 4) and the average module 
size in the system is moderately larger than the average 
size in this sample (factor 19) . 

• Have moderately low availability of resources (fac- 
tor 18), which cannot be adequately supplied in a decaying, 
archaic computing environment. Therefore, the effectiveness 
of the development interface managers is generally low (fac- 
tor 7) . 
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3 • High-Level Factor 3 - High Use of Software Engineering 
Technology (12 percent of variance) . The factors strongly 
correlated with this high-level factor include 

a. Factor 1 (+0.67) --High general use of software en- 
gineering practices (MT, TS, DC, and SE) 

b. Factor 6 (+0.60) — High level of ability of techni- 
cal staff (AP, MG, ,?F, and AB) 

c. Factor 18 (+0 ,58) --High availability of resources 
(RA) 

d. Factor 30 (+0.56) --New project type (WF and PS) 

e. Factor 8 (-0 .61) --Overall instability of managers 
(MG and PF) 

Factors that are moderately correlated include 

a. Factor 2 (+0.50) — Moderately high use of batch de- 
velopment techniques (MT, TS, and DC) 

b. Factor 14 (+0 .52) --Less than average schedule dif- 
ficulty and moderately unresponsive analysis leader 
(CP and EX) 

c. Factor 22 (+0.52) — Moderately high level of appli- 
cable experience and good reputation of technical 
staff (RK, YP, YA, and YE) 

d. Factor 38 (+0.48) — Moderately complex code (MS and 
SW) 

e. Factor 25 (-0 .51) --Moderately low level of environ- 
ment experience of project and analysis managers 
for implementation (YA and YE) 
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This high-level factor represents the high use of software 
engineering technology. In general, software engineering 
practices are used to the highest degree when 

• The technical staff has a moderately high level of 

* ability (factor 6) and a moderately high level of 

applicable experience (factor 22) 

• The software product is mostly new design and code 
(factor 30) 

• The team has reasonable access to resources (fac- 
tor 18) 

This trend occurs even though there is not much stability 
among the interfacing managers and leaders (factors 8 and 
14). 

This trend is not surprising because most programmers do not 
want to work with old code. High performers are more flexi- 
ble, willing to embrace new technologies, able to see the 
benefits of them, and can find ways to obtain resources. 
While it is possible to train personnel and to improve their 
performance, it is not possible to get rid of old code be- 
cause its use is deemed cost effective. The development 
manager's job of training programmers to use modern program- 
ming practices while modifying, enhancing, or rebuilding old 
systems is not an easy one. 

The projects characterized by high use of software technol- 
ogy, high level of personnel ability, and mostly new soft- 
ware tend to 

• Have schedules with less than average difficulty 

(factor 14) "3 

. ; . . : • (/" .. . . ■ . . : . 

• Use primarily batch development techniques (fac- 
tor 2) 
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• Produce moderately complex code (factor 38) com- 
pared with the average complexity in this sample , 
measured by the number of decisions per 1000 execu- 
table LOC 

4 . High-Level Factor 4 - Small Design Team (11 percent of 
variance) . The factors strongly correlated with this high- 
level factor include 

a. Factor 21 (+0 .62) —High degree of formal training. 
The sign of the correlation is reversed because the 
sign of the correlation of the principal measure is 
negative (RA) 

b. Factor 32 (+0.74) — Small design team (WF) 

e. Factor 15 (-0.86) — Very responsive analysis leader 
(EX) 

Factors that are moderately correlated include 

a. Factor 19 (+0.50) — Small average module size. The 
sign of the correlation is reversed because the 
signs of the correlations of the principal measures 
are negative (PR) 

b. Factor 34 (+0 .56) --Large data base size (MS and SW) 

c. Factor 5 (-0 .51) --Moderately low use of chief pro- 
grammer. The sign of the correlation is reversed 
because the signs of the correlations of the prin- 
cipal measures are negative (MT> TS, and DC) 

d. Factor 9 (-0.49) — Ineffectiveness of analysis man- 
agers (AP and MG) 

This high-level factor represents the small design team. In 
general, team members have been formally trained in develop- 
ment (factor 21) and work with systems that have complex in- 
terprogram communications, data base structures, and 
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execution timing constraints (factor 32) . This trend is re- 
enforced by the fact that the systems have moderately large 
data bases (factor 34) . Although there is no correlation 
with system size, this factor indicates moderately low use 
of the chief programmer (factor 5) . The average module size 
tends to be smaller (factor 19) than the average as a result 
of formal training. 

5 . High-Level Factor 5 - Highly Qualified Team (11 percent 


of variance ) . The factors strongly correlated with this 
high-level factor include 

a. Factor 20 (+0.61) — High level of support for sup- 
port software (RA) 

b. Factor 28 (+0.58) -^Highly qualified programmers (WF 
and PS) 

Factors that are moderately correlated include 

a. Factor 6 (+0 .53) --Moderate level of ability of 
technical staff (AP, MG, PF, and AB) 

b. Factor 10 (+0.44) — Moderately effective analysis 
managers (AP and MG) 

c. Factor 31 (+0.48) — Moderate use of code reading and 
moderately less t, a spent during testing phases 
(WF and PS) 

d. Factor 11 (-0.51) --Instability of experienced proj- 
ect manager. The sign of the correlation is re- 
versed because the sign of the correlation of the 
principal measure is negative (MG) 

e. Factor 13 (-0 .48) --Moderately good external support 
(EX) 



f. Factor 16 (-0 .47) --Moderately good high-level de- 
velopment support and moderately high team turnover 
(IN and EX) 

g. Factor 34 (-0 .46) --Small data base size (MS and SW) 

This high-level factor summarizes the qualifications of the 
development team for the application. The characteristics 
of a highly qualified team are highly qualified programmers 
(factor 28), a technical staff with a moderately high level 
of ability (factor 6), an experienced project manager (fac- 
tor 11) , a moderately effective analysis manager (fac- 
tor 10), and a moderately supportive development interface 
manager (factor 16). This development interface manager is 
directly and indirectly responsible for a high level of sup- 
port for support software (factor 20) and hardware (fac- 
tor 16), as well as for simulator data and analysis support 
(factor 13). However, the team is likely to lose its proj- 
ect manager (factor 11) and to have a high turnover of pro- 
grammers (factor 16) . 

4.4.2 UNRELATED CLASS FACTORS 

The following list contains class factors that are not cor- 
related with any of the high-level (hereafter referred to as 
HL) class factors at the 0.01 significance level. The fac- 
tor with which each class factor is most strongly correlated 
and the value of that correlation appear in parentheses fol- 
lowing the class factor number. 

1. Factor 3 (HL factor 5, -0.40) --Low use of top-down 
design techniques (MT) 

2. Factor 5 (HL factor 1, +0.54) — High use of chief 
programmer. The sign is reversed (MT, TS, and DC) 

3 . Factor 9 (HL factor 4, -0 .49) --Ineffectiveness of 
analysis' managers (AP and MG) 
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4. Factor 10 (HL factor 2;, -0 .54) --Ineffectiveness of 
analysis leader (AP and MG) 


5. Factor 11 (HL factor 5, -0.51) — Instability of ex- \\ j 

perienced project manager. The sign is reversed ! . | 

(MG) 

r 

6 • Factor 16 (HL factor 5, -0.47) --Gooc high-level | j 

development support and high team turnover (IN and ! , 

EX) f-. : >1 

| ' i 

7. Factor 19 (HL factor 4, +0.50) — Small average mod* I 

ule size. The sign is reversed (PR) 

8. Factor 22 (HL factor 3, +0.52) --High level of ap- 

; J 

plicable experience and good reputation of techni- j, 

cal staff (RK, YP, YA, and YE) j 



9. Factor 23 (HL factor 2* -0.45) — Low level of pro- 
fessional experience of technical staff (RK* YP* 

• YA* and YE) 

1 . Factor 24 (HL factor 1* +0.44) --Low level of envi- 
es j ronment experience of development managers (RK and 

■ ' \V VE ) 

11. Factor 25 (HL factor 3, -0.51) — Low level of envi- 
ronment experience of project and analysis managers 
for implementation (YA and YE) 

12. Factor 29 (HL factor 1* +0.47) — Tendency to use 
graphics test time rather than batch testing meth- 
ods (WF) 

13. Factor 34 (HL factor 4 * +0 .56) --Large data base 
size (MS and SW) 

14. Factor 35 (HL factor 1* +0.48) — Large number of, 
data sets for internal communication (MS and SW) 


Factor 36 (HL factor 4, +0.44) --High use of old 
code (SW) 




15. 


16. Factor 37 (HL factor 1, -0 .49) --Small number of 

errors in code. The sign is reversed (SW) 

17. Factor 38 (HL factor 3, +0 .48) --Complex code (MS 

and SW) 

Factors 3, 24, and 36 are the only class factors not corre 
lated with any high-level factor at the 0.05 significance 
level. 




SECTION 5 - CONCLUSIONS 


This section summarizes the conclusions derived from the 
analysis of the SEL software management measures collected 
for 20 independent software development projects. Sec- 
tion 5.1 summarizes the findings stated either explicitly or 
implicitly in Section 4, and Section 5.2 recommends a course 
of action based on these findings. 

5.1 FINDINGS 

The findings are presented below in the order in which they 
were derived by the analytical procedure (see Figure 3-1). 

Test of Normality . This test shows that not all measures are 
appropriate; that is, some measures fail to differentiate one 
group of projects from another. Approximately one-half of 
the measures rejected by the test of normality are measures 
that were developed for another environment or for use with 
several environments. The results of additional analyses of 
SEL measures will not change their status. 

The development managers should discard these measures un- 
less there are 

• Anticipated changes in the development process and 
environment 

• Plans for expansion of the organization's line of 
business 

• Intentions to combine SEL data with data from other 
environments that use the same or similar measures 

Factor Analysis of the Seven Classes of Measures. This 


analysis 

1. Shows that by producing descriptive factors, a 
large set of measures can be reduced by approximately an 
order of magnitude. ) - 






2. Provides a basis for selecting the most important 
measures by their correlations with the descriptive factors 
and the ease with which measurements can be made. 

3. Shows that many measures are related to system 
size. Apparently, whether consciously or not, managers tend 
to evaluate various effects with respect to system size (for 
example, complexity ratings). 

Cluster Analysis of the Seven Classes of Measures . This 
analysis 

1. Shows that the major distinction made among proj- 
ects by the factors is system size and that System 730 was 
misclassif ied as a large project. 

2. Confirms that the Walston-Felix (WF) category ;of 

measures is an excellent set of measures and that, to a 
large extent, the SEL measures are basically an expansion 
and elaboration of the WF measures. The reader can see this 
by comparing Figure 4-1 (cluster map for all screened SEL 
measures) with Figure A. 6. 1-2 in Appendix A of Volume 2 
(cluster map for WF category) . The comparison shows that 
the cluster maps are very nearly identical in project pair- 
ings, although differences in the levels of similarity 
exist. There is one exception to this generalization: Sys- 

tem 200' s high degree of similarity with System 900 rather 
than with System 300. 

Because the cluster maps (see Figure 4-1) of the small proj- 
ect group and the large project group, separately, are not 
similar to a cluster map of any other individual category of 
measures, the levels of similarity within the two groups are 
a reflection of the effects of all classes of measures, 
i.e. , software engineering, experience, complexity, and 
product effects. The differences in the levels of similar- 
ity indicated by comparison of the cluster map in Figure 4-1 
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and the cluster map for the WF measures (see Figure A. 6.1-2) 
result from the finer detail added by the SEL measures. 

Factor Analysis of the 38 Class Factors . This analysis 

1. Shows that the major distinction made among proj- 
ects is still system size; however , experience-related ef- 
fects are obvious. 

2. Points out subtle environmental quirks and weak- 
nesses and, therefore, can alert development managers to 
take preventive steps, plan better for contingencies, and 
generally improve the process through proper training, Two 
of the important effects are as follows; 

a. The high use of software engineering technol- 
ogy is strongly correlated with the level of ability of 
the technical staff and the degree to which a system is 
new, i.e., amount of new design and code. An obvious re 
lationship of high technology use with team ability has 
been discussed within the SEL over the past few years. 
However, the relationship with the degree of system new- 
ness had not been apparent. The desire of programmers 
to completely develop something on their own (i.e., not 
constrained by large amounts of existing code) so that 
they can improve their skills is a familiar lament when 
assignments are made. Even though sound software engi- 
neering methodology is taught and its use is encouraged 
and enforced, programmers are apparently more highly 
motivated to use the methodology when they are satisfy- 
ing career objectives. 

In this environment, the development manager is faced 
with a difficult challenge in training personnel to use 
better technologies and in motivating them to use the 
technologies for maintenance, enhancement, and rebuild- 
ing activities where fairly large amounts of reused code 
are involved. 


b. Although some current thinking is to use 
highly qualified teams to make large improvements in 
productivity and/or reliability, in this environment, 
highly qualified teams are likely to be unstable because 
experienced project managers and leaders are likely to 
be promoted and because career-minded programmers do not 
want to work in the same computing environment or with 
the same application very long. 

Therefore, in this environment, higher level managers 
will have to be very creative in forming highly quali- 
fied teams and in helping team members to set career 
goals. Otherwise, team members are likely to leave the 
project for one reason or another to improve their 
skills, and the very purpose of bringing such a team 

a „ 

- together is defeated. 

3. Shows that the data are not ideally represented for 
determining what is related to productivity or reliability. 
For example, only the Development Team Background class fac- 
tor that accounts for the least amount of the variance 
within the class (factor 26) is strongly correlated with any 
of’ the combined factors, although all five class factors are 
moderately correlated with the combined factors. Years of 
experience (YOE) , which obviously has a direct bearing on 
the outcome of a software development process, can be repre- 
sehted better. For example, an average person with 25 YOE 
is not likely to be 25 times better or faster than an aver- 
age person with 1 YOE; however, he/she may be 7 times better 
or faster. No attempt to find a suitable transformation for 
years of experience or to normalize system size in terms of 
effort or errors was planned for this analysis; however, it 
is planned for future analysis 


Cluster Analysis of the 38 Class Factors . This analysis 

1. Shows that the major distinction made among proj- 
ects by the factors is still system size. The preservation 
of this distinction, in spite of the dilution of the effect 
of explicit size measures (Section 4.3), suggests that the 
effect of size is implicit in many not obviously related 
measures. For example, more experienced personnel may be 
regularly assigned to larger projects. Thus, team experi- 
ence would be related to project size, although not obvi- 
ously so. 

2. Shows that the analytical technique reduces con- 

founding effects (primarily overwhelming information in 
terms of system size) and points out a second major distinc- 
tion among the small and the large projects: experience. 

The reader can see this by comparing the small project group 
in Figure 4-2 (cluster map for combined factors) with Fig- 
ure A. 5. 3-4 in Appendix A of Volume 2 (cluster map of Years 
of Applicable Experience (YA) category) . Excluding Sys- 
tem 730, which was misclassif ied, comparison with the 
cluster maps of each category shows that the YA category has 
the same project pairings, considering small differences in 
the levels of similarity. This is encouraging because the 
outcome of a small development project is considered by many 
to be primarily related to the developers' experience. 

The reader can see the experience relationship among the 
large projects by comparing the large project group in Fig- 
ure 4-2 (cluster map for combined factors) with Fig- 
ure A. 2.1-3 in Appendix A of Volume 2 (cluster map of 
Experience With Application (AP) category) . Excluding Sys- 
tem 730, which was misclassif ied, comparison with the 
cluster maps of each category shows that the AP category is 
very nearly identical in project pairings, although there 
are some differences in levels of similarity. 


The fact that different kinds of experience measures differ- 
entiate the small projects from the large projects is ex- 
plained as follows. In general, smaller projects are 
usually staffed with more junior personnel in the environ- 
ment; that is, they have little experience with the applica- 
tion, but they have varying degrees of applicable 
experience. Larger projects, however, are staffed with per- 
sonnel who also have varying degrees of applicable experi- 
ence and varying degrees of experience with the application. 

The differences within the small and large project groups in 
the cluster map in Figure 4-2 are primarily because of ex- 
perience levels. The differences between the cluster map in 
Figure 4-1, which closely resembles the cluster map for the 
Walston-Felix measures, and the cluster map in Figure 4-2 is 
a result of the finer detail in experience data provided by 
the SEL measures after the effects of system size have been 
reduced. 

$.2 RECOMMENDATIONS 

The analysis described in Section 4 and summarized in Sec- 
tion 5.1 identifies a number of significant software devel- 
opment qualities (or factors) . That analysis evaluates the 
effectiveness of a large set of management measures with 
respect to those qualities. Thus, a basis has been estab- 
lished for defining a more concise set of software develop- 
ment measures. The measures considered cover every aspect 
of software development experienced by the SEL. 


Although the complete set of 608 measures could be collected 
from every software development effort monitored, that proc- 
ess would be a tedious and time-consuming (expensive) proc- 
ess. The foregoing analysis demonstrates that a smaller set 
of measures oriented toward the basic qualities (factors) 
present in the data can be found. This smaller set contains 
most of the information of the original measures. A set of 
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38 appropriately selected measures retains about 80 percent 
of the information content of the complete set of 608. 

Tables 5-1 through 5-7 identify the elements of such a con- 
cise set of measures. The measure (s) corresponding to each 
factor in the table is, in most cases, the measure most 
highly correlated with it. However, another highly corre- 
lated measure was sometimes selected because it appeared 
more representative or easier to obtain. Additional meas- 
ures are included in the tables where additional information 
is desirable. A total of 78 measures is listed. The set of 
measures listed in these tables, then, constitutes the SEL's 
recommendation of management measures that should be col- 
lected and monitored during software development. : 

Figure 5-1, at the end of this section, demonstrates the ac- 
curacy of this set of 78 important measures. The cluster 
map shown in the figure preserves the size effect previously 
noted (see Sections 4.3 and 5.1), although it provides more 
detail. Note that three clusters are defined that corre- 
spond to large, intermediate* and small systems. The inter- 
mediate cluster is composed of the three smallest of the 
large systems. Furthermore, transpositions of the order of 
systems in the graph can be made (while preserving the re- 
lationships among the systems), which would produce a rank- 
ing of the systems from largest to smallest in terms of 
developed lines of code, with one exception. Thus, not only 
is project size a major characteristic, but, based on this 
analysis, developed lines of code is the most important 
measure of a software project of this application type in 
this environment. 

These recommendations should not, however, be construed to 
imply that the study of software measures is complete. The 
SEL and other researchers are still engaged in an extensive 
review and analysis of measures. The study documented in 






J" ' ■ 


this report considers only one set of measures. Further- 
more, it does not analyze all the interrelationships of 
these measures or their specific applications. Those issues 
will be considered in a future document (Reference 13) . 
However, this document does show how a relatively small set 
of measures can be defined in such a manner as to fully 
characterize or describe the software development process 0 
experienced by the SEL. 





Table 5-2. Important Measures in the Development Team Ability Class 
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Table 5-3. Important Measures in the Difficulty of Project Class 
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Table 5-4. Important Measures in the Process and Product Characteristics Class 
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Figure 5-1 . Cluster Map Based on Important Measures 



APPENDIX A * FACTOR ANALYSES OP CLASSES OP MEASURES 

- — - " 1 ii , IIU ." ' ‘ l J 1 '. ll . 

This appendix reproduces the output of factor analyses of 
each of the classes of measures defined in Section 2.2. 
These classes are as follows: 


1. Software engineering practices (Table A-l) 

2, Development team ability (Table A-2) 

3* Difficulty of project (Table A-3) 

4. Process and product characteristics (Table A-4) 

5. Development team background (Table A- 5) 

6. Resource model parameters (Table A-6) 

7. Additional detail (Table A-7) 


The data used is that of the 20 independent systems de- 
scribed in Section 2.1. The output of the factor analysis 
procedure includes three types of information that are es- 
sential to interpreting its results. These information 
types are 

• Factor loading-~percentage of the variance in the 
data accounted for by each factor; shows the rela- 
tive importance of factors 

• Factor pat tern- -cor relations of all measures with 
all factors; shows the underlying structure of the 

data ■ "■■■.■. : 

i! . , 

• Communality — percentage of each measure’s variance 
accounted for by all factors; shows how well each 
measure, is explained by the factor model 

The information presented in the seven tables (one for each 
class of measures) is divided into three subtables (one for 
each type of information) . The statistical software used to 
generate these tables' is, described in Reference 7, 
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Table A-lb. Factor Analysis of Software Engineering 
Factor Pattern 
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Table A-2a . Factor Analysis of Development Team Ability: 
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Table A— 2b. Factor Analysis of Development Team Ability: 
Factor Pattern (1 of 3) 
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Table A-2b. Factor Analysis of Development Team Ability 
Factor Pattern (2 of 3) 
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Table A-7b. Factor Analysis of Additional Detail 
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APPENDIX B - FACTOR ANALYSIS OF CLASS FACTORS 


Table B-l reproduces the output of the factor analysis of 
the 38 class factors defined in Section 4.2. Appendix A 
(Volume 1) reproduces the analyses that generated the class 
factors analyzed in Table B-l. The elements of Table B-l 
are explained in Section 3.2 and Appendix A of this volume . 
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